Turning Scan Reports into JSON Is Solved. Spotting Contradictions Isn't
Imaging departments hold years of free-text scan reports. Registries, audits and research projects need the same reports as structured fields: which tracer, how much activity, how long after injection, and for each lesion its site, side, size and uptake.
Doing that by hand is slow, and the reports often can’t leave the hospital network. Task L2 asks whether a model on a desk-side box can do the conversion unattended, and whether it notices when a report contradicts itself.
Yes, for all five local models. Every model scored a field F1 of at least 0.98 with 100% valid JSON. Qwen3.8-27B and Gemma 4 matched Claude Opus 5.5's perfect 1.000. On this set, extraction is solved.
The one signal that separates the models is contradiction detection. Opus and DeepSeek-V4.1-Flash caught every planted contradiction. Qwen3.8 and Gemma 4 caught 88%, Qwen3.6 82%, Qwen3-Coder-Next 65% and Nemotron 3.5 only 18%.
The task in one picture
The model reads one report and fills a fixed JSON form. The server constrains its output to that form, so it can’t produce broken JSON.
A few terms. SUVmax is the peak “standardised uptake value” in a lesion: how strongly it takes up the tracer, the main number a PET reader reports. F1 combines precision (the fields the model filled are right) and recall (it filled every field it should), on a 0–1 scale. Grammar-constrained decoding means the server only lets the model emit tokens that keep the output valid against the schema.
The data
Real reports weren’t available: access to the planned set of PET reports was not obtained. So a generator (fixed seed, so anyone can rebuild it) works backwards. It draws the structured record first, then writes it up as varied free text.
| What | Detail |
|---|---|
| Reports | 300 synthetic PET/CT and SPECT reports |
| Fields per report | study type, tracer, activity, uptake time, blood glucose, liver SUVmean, 0–5 lesions (site, side, size, SUVmax), impression category |
| Variation | MBq or mCi, mm or cm, three section-title styles, three phrasings per finding, optional sign-off |
| Omissions | 12 reports leave out a field (the right answer is null) |
| Contradictions | 17 reports state one value twice with different numbers |
| Prompt examples | 2 worked examples, generated with a different seed |
The planted contradictions are the interesting part. The model is told to
report the first-stated value and set "contradiction": true. The
contradicted field isn’t scored, so the flag is the only test.
What a call looks like
The system prompt holds the rules and the schema. The key rules: convert mCi to MBq and cm to mm, use null for anything not stated, and flag a contradiction when the report gives two different values for the same quantity. Here is one of the planted ones (report r2026-177; all reports are synthetic):
Bone SPECT/CT REPORTIndication: restaging after therapy.
TECHNIQUE The patient received 21.4 mCi (791 MBq) 99mTc-MDP IV. Uptake time 166 min. SPECT/CT acquired over the region of interest. Scan start logged at 186 min after injection.
Findings: No focal abnormal uptake.
IMPRESSION No pathological uptake identified.
{"study_type": "Bone SPECT/CT", "radiopharmaceutical": "99mTc-MDP",
"administered_activity_mbq": 791, "uptake_time_min": 166,
"blood_glucose_mmol_l": null, "liver_suvmean": null, "lesions": [],
"impression_category": "no abnormal uptake", "contradiction": true}Identical to the gold, including "contradiction": true.
Identical except "contradiction": false.
The two values are written in different words, “Uptake time 166 min” and “Scan start logged at 186 min after injection”. To flag it, the model has to see that both describe the same interval. Every model extracted the fields correctly here; only the flag differs. Three of the planted cases show the pattern:
| Contradiction | Opus | Qwen3.8 | Gemma 4 | Qwen3.6 | Coder-Next | Nemotron |
|---|---|---|---|---|---|---|
| r2026-177 (166 vs 186 min) | yes | yes | yes | yes | no | no |
| r2026-235 (46 vs 62 min) | yes | yes | yes | yes | no | no |
| r2026-116 (559 vs 499 MBq) | yes | yes | yes | yes | yes | no |
Results: a ceiling on extraction, a spread on flags
The pass rule was fixed before any model ran: field F1 of at least 0.90, 100% valid JSON, and within 3 points of Opus. Each lesion attribute and each scalar field counts as one item. Lesions are paired with the gold by Hungarian matching, a standard algorithm that finds the best one-to-one pairing, so a missed lesion costs its four attributes.
Field F1
300 reports, median of 3 runs for local models. Scale 0–1; every bar is at or near the ceiling.
Planted contradictions caught
Recall on the 17 reports that state one value twice. Scale 0–100%. No pass bar was set for this signal.
| Model | Field F1 | Valid JSON | Contradictions caught | False alarms | Decode tok/s | $ / 1k reports | Verdict |
|---|---|---|---|---|---|---|---|
| Opus 5.5 | 1.0000 | 100% | 1.000 | 0.000 | – | 17.36 | baseline |
| DeepSeek-V4.1-Flash | 0.9992 | 100% | 1.000 | 0.004 | – | 0.91 | reference |
| Qwen3.8-27B | 1.0000 | 100% | 0.882 | 0.000 | 20.2 | 0.020 | small model ready |
| Gemma 4 26B-A4B | 1.0000 | 100% | 0.882 | 0.004 | 22.7 | 0.016 | small model ready |
| Qwen3.6-35B-A3B | 0.9972 | 100% | 0.824 | 0.007 | 28.9 | 0.011 | small model ready |
| Qwen3-Coder-Next | 0.9982 | 100% | 0.647 | 0.007 | 23.6 | 0.015 | small model ready |
| Nemotron 3.5 30B-A3B | 0.9800 | 100% | 0.176 | 0.018 | 26.7 | 0.013 | small model ready |
False alarms are the share of the clean reports wrongly flagged. Decode speed is per stream with four reports in flight. Qwen3.8-27B and Gemma 4 score 1.000 on all 11 fields, and their three sampled runs agree on every report.
What the audit found
We read every mismatch in run 0 of Qwen3.6 and Nemotron against the report text. The scorer was right every time. The errors are real but rare:
- Qwen3.6 merged two lesions in report r2026-041. The report has a breast focus of 8 mm with SUVmax 4.3, then a left-breast focus of 8 mm with SUVmax 6.1. Qwen3.6 kept only the second and set the contradiction flag, reading the two lines as one lesion stated twice. That’s wrong against the gold, but an understandable reading of an awkward report. The missing lesion’s four attributes count as misses.
- Nemotron put one lesion’s size on another lesion in report r2026-010.
The audit also found two flaws in the generator. They affect every model alike and cost at most about a point:
- Impressions don’t fit the study. The impression is drawn independently of the study type, so a heart perfusion scan can read “concerning for malignant disease”. Models that answered “indeterminate” were marked wrong against an implausible gold.
- Midline organs get a side. The generator writes things like “left peritoneum”. Models that left the side null lost one attribute.
What this means if you structure reports
- Use a schema and grammar-constrained decoding. With both, every local model produced valid JSON on every report.
- Read “small model ready” as “extraction isn’t the bottleneck”. Template text is far cleaner than real dictation. Hand-structure about 50 of your own reports and check before you rely on it.
- For consistency flags, use Qwen3.8-27B or Gemma 4, and still expect to miss about 1 contradiction in 8. Don’t rely on Nemotron for flags (18%).
- Cost is not a reason to stay on a large model. Locally, 1,000 reports cost 1–2 cents, against $17.36 on Opus and $0.91 on DeepSeek.
Limits. The reports are synthetic and template-rendered, so the set is at its ceiling and doesn't separate the models on extraction. The planned check on real reports wasn't possible, because access to them was not obtained. Opus's cost is inflated by its always-on thinking. DeepSeek's API has no schema-constrained decoding, so it got the schema in the prompt with JSON mode instead.
