Small Models on One DGX Spark

Turning Scan Reports into JSON Is Solved. Spotting Contradictions Isn't

Two bars per model: extraction accuracy all at the ceiling, contradiction catches falling from 100% to 18%

Imaging departments hold years of free-text scan reports. Registries, audits and research projects need the same reports as structured fields: which tracer, how much activity, how long after injection, and for each lesion its site, side, size and uptake.

Doing that by hand is slow, and the reports often can’t leave the hospital network. Task L2 asks whether a model on a desk-side box can do the conversion unattended, and whether it notices when a report contradicts itself.

Small model ready L2 verdict

Yes, for all five local models. Every model scored a field F1 of at least 0.98 with 100% valid JSON. Qwen3.8-27B and Gemma 4 matched Claude Opus 5.5's perfect 1.000. On this set, extraction is solved.

The one signal that separates the models is contradiction detection. Opus and DeepSeek-V4.1-Flash caught every planted contradiction. Qwen3.8 and Gemma 4 caught 88%, Qwen3.6 82%, Qwen3-Coder-Next 65% and Nemotron 3.5 only 18%.

The task in one picture

The model reads one report and fills a fixed JSON form. The server constrains its output to that form, so it can’t produce broken JSON.

InputOne free-text reportPET/CT or SPECT, written in varied styles and units
Local modelFill the schemaRules, schema and 2 worked examples in the prompt; output constrained by a JSON grammar
CodeMatch and compareEach field and lesion attribute checked against the known answer
MetricsF1, validity, flagsField F1, valid JSON, contradictions caught, false alarms

A few terms. SUVmax is the peak “standardised uptake value” in a lesion: how strongly it takes up the tracer, the main number a PET reader reports. F1 combines precision (the fields the model filled are right) and recall (it filled every field it should), on a 0–1 scale. Grammar-constrained decoding means the server only lets the model emit tokens that keep the output valid against the schema.

The data

Real reports weren’t available: access to the planned set of PET reports was not obtained. So a generator (fixed seed, so anyone can rebuild it) works backwards. It draws the structured record first, then writes it up as varied free text.

What Detail
Reports 300 synthetic PET/CT and SPECT reports
Fields per report study type, tracer, activity, uptake time, blood glucose, liver SUVmean, 0–5 lesions (site, side, size, SUVmax), impression category
Variation MBq or mCi, mm or cm, three section-title styles, three phrasings per finding, optional sign-off
Omissions 12 reports leave out a field (the right answer is null)
Contradictions 17 reports state one value twice with different numbers
Prompt examples 2 worked examples, generated with a different seed

The planted contradictions are the interesting part. The model is told to report the first-stated value and set "contradiction": true. The contradicted field isn’t scored, so the flag is the only test.

What a call looks like

The system prompt holds the rules and the schema. The key rules: convert mCi to MBq and cm to mm, use null for anything not stated, and flag a contradiction when the report gives two different values for the same quantity. Here is one of the planted ones (report r2026-177; all reports are synthetic):

Input report
Bone SPECT/CT REPORT

Indication: restaging after therapy.

TECHNIQUE The patient received 21.4 mCi (791 MBq) 99mTc-MDP IV. Uptake time 166 min. SPECT/CT acquired over the region of interest. Scan start logged at 186 min after injection.

Findings: No focal abnormal uptake.

IMPRESSION No pathological uptake identified.

Gold
{"study_type": "Bone SPECT/CT", "radiopharmaceutical": "99mTc-MDP",
 "administered_activity_mbq": 791, "uptake_time_min": 166,
 "blood_glucose_mmol_l": null, "liver_suvmean": null, "lesions": [],
 "impression_category": "no abnormal uptake", "contradiction": true}
Qwen3.8-27B flag caught

Identical to the gold, including "contradiction": true.

Nemotron 3.5 flag missed

Identical except "contradiction": false.

The two values are written in different words, “Uptake time 166 min” and “Scan start logged at 186 min after injection”. To flag it, the model has to see that both describe the same interval. Every model extracted the fields correctly here; only the flag differs. Three of the planted cases show the pattern:

Contradiction Opus Qwen3.8 Gemma 4 Qwen3.6 Coder-Next Nemotron
r2026-177 (166 vs 186 min) yes yes yes yes no no
r2026-235 (46 vs 62 min) yes yes yes yes no no
r2026-116 (559 vs 499 MBq) yes yes yes yes yes no

Results: a ceiling on extraction, a spread on flags

The pass rule was fixed before any model ran: field F1 of at least 0.90, 100% valid JSON, and within 3 points of Opus. Each lesion attribute and each scalar field counts as one item. Lesions are paired with the gold by Hungarian matching, a standard algorithm that finds the best one-to-one pairing, so a missed lesion costs its four attributes.

Field F1

300 reports, median of 3 runs for local models. Scale 0–1; every bar is at or near the ceiling.

Small modelLarge-model reference0.90 bar
Opus 5.51.000
DeepSeek V4.1 Flash0.999
Qwen3.8-27B1.000
Gemma 41.000
Qwen3-Coder-Next0.998
Qwen3.60.997
Nemotron 3.50.980
00.250.500.751.00

Planted contradictions caught

Recall on the 17 reports that state one value twice. Scale 0–100%. No pass bar was set for this signal.

Small modelLarge-model reference
Opus 5.5100%
DeepSeek V4.1 Flash100%
Qwen3.8-27B88.2%
Gemma 488.2%
Qwen3.682.4%
Qwen3-Coder-Next64.7%
Nemotron 3.517.6%
0%25%50%75%100%
Model Field F1 Valid JSON Contradictions caught False alarms Decode tok/s $ / 1k reports Verdict
Opus 5.5 1.0000 100% 1.000 0.000 – 17.36 baseline
DeepSeek-V4.1-Flash 0.9992 100% 1.000 0.004 – 0.91 reference
Qwen3.8-27B 1.0000 100% 0.882 0.000 20.2 0.020 small model ready
Gemma 4 26B-A4B 1.0000 100% 0.882 0.004 22.7 0.016 small model ready
Qwen3.6-35B-A3B 0.9972 100% 0.824 0.007 28.9 0.011 small model ready
Qwen3-Coder-Next 0.9982 100% 0.647 0.007 23.6 0.015 small model ready
Nemotron 3.5 30B-A3B 0.9800 100% 0.176 0.018 26.7 0.013 small model ready

False alarms are the share of the clean reports wrongly flagged. Decode speed is per stream with four reports in flight. Qwen3.8-27B and Gemma 4 score 1.000 on all 11 fields, and their three sampled runs agree on every report.

≈ 2.8 hGPU time, 6.6k local calls
~7 sper report at 4 streams (~1,600 tokens in, ~140 out)
$0.011–0.020per 1,000 reports locally, vs $17.36 on Opus and $0.91 on DeepSeek

What the audit found

We read every mismatch in run 0 of Qwen3.6 and Nemotron against the report text. The scorer was right every time. The errors are real but rare:

  • Qwen3.6 merged two lesions in report r2026-041. The report has a breast focus of 8 mm with SUVmax 4.3, then a left-breast focus of 8 mm with SUVmax 6.1. Qwen3.6 kept only the second and set the contradiction flag, reading the two lines as one lesion stated twice. That’s wrong against the gold, but an understandable reading of an awkward report. The missing lesion’s four attributes count as misses.
  • Nemotron put one lesion’s size on another lesion in report r2026-010.

The audit also found two flaws in the generator. They affect every model alike and cost at most about a point:

  • Impressions don’t fit the study. The impression is drawn independently of the study type, so a heart perfusion scan can read “concerning for malignant disease”. Models that answered “indeterminate” were marked wrong against an implausible gold.
  • Midline organs get a side. The generator writes things like “left peritoneum”. Models that left the side null lost one attribute.

What this means if you structure reports

  1. Use a schema and grammar-constrained decoding. With both, every local model produced valid JSON on every report.
  2. Read “small model ready” as “extraction isn’t the bottleneck”. Template text is far cleaner than real dictation. Hand-structure about 50 of your own reports and check before you rely on it.
  3. For consistency flags, use Qwen3.8-27B or Gemma 4, and still expect to miss about 1 contradiction in 8. Don’t rely on Nemotron for flags (18%).
  4. Cost is not a reason to stay on a large model. Locally, 1,000 reports cost 1–2 cents, against $17.36 on Opus and $0.91 on DeepSeek.

Limits. The reports are synthetic and template-rendered, so the set is at its ceiling and doesn't separate the models on extraction. The planned check on real reports wasn't possible, because access to them was not obtained. Opus's cost is inflated by its always-on thinking. DeepSeek's API has no schema-constrained decoding, so it got the schema in the prompt with JSON mode instead.