Skip to content
NutsNews chestnut logoNutsNews Wiki

Technical guide · Core platform

Page status: Draft

NutsNews summary screening: Qwen 2.5 3B versus 7B

Historical screening results and their production limitations.

Visual overview

Primary diagram

System map

A small controlled model screening provides observations, but production stays unchanged until representative repeated chat tests and human review support a change.

Render the repository-owned system map when you need it.

Diagram is not rendered yet.

View as text
NutsNews summary screening: Qwen 2.5 3B versus 7B decision

A small controlled model screening provides observations, but production stays unchanged until representative repeated chat tests and human review support a change.

flowchart TD
  accTitle: NutsNews summary screening: Qwen 2.5 3B versus 7B decision
  accDescr: A small controlled model screening provides observations, but production stays unchanged until representative repeated chat tests and human review support a change.
  inputs["Three source-checked story digests"] --> test["One controlled run per story and model"]
  test --> results["Record speed, length and factual observations"]
  results --> limits["Small sample; not the production chat workflow"]
  limits --> decision["Keep production unchanged"]
  decision --> future["Repeated representative chat tests and human review"]

NutsNews summary screening: Qwen 2.5 3B versus 7B decision

Fullscreen diagram view.

NutsNews summary screening: Qwen 2.5 3B versus 7B

Section titled “NutsNews summary screening: Qwen 2.5 3B versus 7B”

Production remains on its existing model. September 13, 2026.

Scope and controls

Section titled “Scope and controls”

Three real stories linked by NutsNews were reduced to source-checked factual digests. Each model received the same digest and the exact buildPrompt function read from the deployed local AI service. Tests used Ollama generate, JSON output, temperature 0, seed 42, 8,192 context capacity, 220 output tokens maximum and two CPU threads, once per story. This is not an exact production replay: production uses chat with an additional system message and its own sampling configuration, ingestion excerpts and output normalization. These results are a screening signal, not sufficient evidence to replace a model or measure general accuracy. Prompts were approximately 400 tokens; no production publishing was triggered.

Sources:

Results

Section titled “Results”
Story3B seconds7B seconds3B characters7B characters
NASA19.3638.75221150
Mouse study9.6517.13205185
Lost toy9.4116.64208155

3B generated approximately 14.7 tokens/sec; 7B approximately 7.0. First requests include loading. Prefix caching and live traffic affect timings. Both returned valid JSON and accepted all three positive stories. 3B met the 200–250 character requirement 3/3 times; 7B 0/3. All 7B decision reasons were empty. The raw result field pass only checks acceptance, not overall quality.

Manual factual review: both retained the mouse-study setting without claiming proven human longevity, but omitted the explicit need for human clinical confirmation. 3B distorted the toy being swept up with bedding into being mistaken for bedding. 7B’s toy summary blurred staged photos into the toy being found enjoying the hotel. 3B’s NASA summary included participation methods; 7B omitted those details. No clear 7B quality advantage was demonstrated on this small sample.

NutsNews health requests all returned 200 during the runs. This does not measure publishing queue throughput or article-generation latency. No shared configuration changed; the test 7B model was unloaded afterward.

Follow-up

Section titled “Follow-up”

Keep the existing model. First improve summary validation and remove generic length-padding behavior: deployed normalizeAcceptedSummary adds filler when text is short, which can reduce specificity independently of model quality. A proposed change should preserve complete sentences and source limitations, and be tested before publishing. No such production change was made in this benchmark. A future model decision needs repeated production-chat replays with the same excerpts and normalization, varied article categories, and human review.