Rarely Wrong, Often Incomplete: Small Models Answering from Clinical Guidelines
Technologists and physicians in a nuclear-medicine department look up dosing, patient preparation and safety rules in procedure guidelines many times a day. An assistant that answers from those guidelines would save time, and running it locally would keep every question inside the hospital.
The bar is strict, though. An answer must carry every key fact, because the caveat attached to a number is often the safety part. The assistant must say so when the guideline doesn’t cover a question. And a confident wrong dose rules a model out. Task L3 asks whether any local model clears that bar.
Not yet. The best local model, Qwen3.8-27B, fully answered 81.7% of the answerable questions, against 90.8% for Claude Opus 5.5. The bar was 85% and within 5 points of Opus.
The gap isn't wrong answers. Local models are wrong about as rarely as the references, around 5% of answers. They lose on completeness: they give the number that was asked for and drop the condition attached to it. Only Qwen3.8-27B never gave a confident wrong dose.
The task in one picture
This is retrieval-augmented generation (RAG): a search step finds the relevant guideline passages, and the model answers from those passages only. We froze the search results, so the model is the only thing that changes between runs.
The data
The corpus is 12 open-access procedure guidelines from the EANM and SNMMI, the European and North American nuclear-medicine societies, split into passages of about 512 tokens. The guideline texts aren’t reproduced here or committed to the study’s repository.
| What | Detail |
|---|---|
| Questions | 150: 120 answerable, 30 unanswerable (10 and 2–3 per guideline) |
| Dose questions | 70 of the 150 |
| Each question has | a reference answer, the key facts it must contain, grading notes |
| Written by | three authoring agents, then checked in a separate pass: 123 ok, 21 edited, 6 replaced |
| Retrieval | the right passage is in the top 8 for 110 of the 120 answerable questions |
The study plan asked for questions written by department staff; agent-written questions are a deviation. The unanswerable questions are ones the guideline leaves to national rules or doesn’t address, and the right reply is to say so.
How answers are graded. A blind judge, DeepSeek-V4.1-Flash, sees the question, the reference answer, the key facts, the grading notes and the answer with the model’s name removed. It scores 2 when every key fact is right, 1 for partly right, and 0 for wrong. Claude Opus 5.5 also graded Opus’s own 150 answers before its API credit ran out, and the two judges gave the same verdict on 96.7% of them.
What a call looks like
Each call carries the question and the eight frozen passages. The system prompt tells the model to answer only from the passages, give exact numbers and units, and never guess a dose. Two examples show the two ways answers go wrong. The passages themselves are guideline text and aren’t shown.
We image at 3 min per bed position on a scanner with 30% or less bed overlap. Using the EANM linear formula, what's the minimum FDG activity for a 70 kg adult?
FDG (MBq) = 14 MBq·min·bed⁻¹·kg⁻¹ × weight (kg) / min per bed
= 14 × 70 / 3 ≈ 327 MBq minimumBoth write out the working, 14 × 70 / 3 = 980 / 3 ≈ 326.7 MBq, and answer about 327 MBq.
Gemma 4: 293.33 MBq. Qwen3-Coder-Next: 343 MBq. Qwen3.6: 280 MBq. Nemotron 3.5: 1470 MBq.
Every model cited the right passage, and the formula and inputs are both in it. The four wrong answers skipped the arithmetic and stated a number anyway. These count as confident wrong dose answers, the failure the safety rule exists to catch.
The second example is the more common failure. Question B021 asks which range of fixed iodine-131 activities is most often used to treat Graves’ disease. The key facts are the range, 370–555 MBq, and the note that a dosimetric approach should be considered for thyroid volumes above 40 ml.
| Model | Answer (run 0) | Score |
|---|---|---|
| DeepSeek-V4.1-Flash | The 370–555 MBq range, plus the dosimetry note for glands above 40 ml | 2 |
| Qwen3.8-27B, Qwen3.6 | “370 to 555 MBq” | 1 |
| Gemma 4 | The same range, in one sentence, without the dosimetry note | 1 |
The local answers are correct and incomplete. A reader who only needed the number is fine; a reader with a large gland misses the part that matters.
The unanswerable questions went well. One asks for the dose-rate limit a patient must be below before discharge after lutetium-177 PSMA therapy, which the guideline leaves to national legislation. All seven models said the passages don’t give a number.
Results
The pass rule, fixed before any model ran: at least 85% fully correct, at least 90% correct abstention on unanswerable questions, zero confident wrong dose answers in any run, and within 5 points of Opus.
Answerable questions answered fully correctly (score 2)
120 answerable questions; median of 3 runs for local models. Scale 0–100%.
| Model | Fully correct | Correct when right passage retrieved | Correct abstention | Wrong doses, worst run | Δ vs Opus | Verdict |
|---|---|---|---|---|---|---|
| Opus 5.5 | 90.8% | 98.2% | 100% | 0 | – | baseline |
| DeepSeek-V4.1-Flash | 90.0% | 96.4% | 100% | 0 | −0.8 | reference |
| Qwen3.8-27B | 81.7% | 88.2% | 100% | 0 | −9.2 | needs a large model |
| Qwen3-Coder-Next | 80.0% | 85.5% | 96.7% | 3 | −10.8 | needs a large model |
| Gemma 4 26B-A4B | 79.2% | 86.4% | 100% | 2 | −11.7 | needs a large model |
| Qwen3.6-35B-A3B | 75.8% | 81.8% | 100% | 1 | −15.0 | needs a large model |
| Nemotron 3.5 30B-A3B | 71.7% | 76.4% | 73.3% | 12 | −19.2 | needs a large model |
Abstention is the one part every model but Nemotron passes. Nemotron asserted an answer on 27% of the unanswerable questions and gave 12 confident wrong doses in its worst run.
Wrong as rarely, but less complete
Splitting the scores into wrong, partial and correct shows where the gap is.
How the answers fail
Share of the 120 answerable questions, pooled over the sampled runs (references: 1 run). The axis runs from 0 to 25%.
The wrong-answer rate is flat: 4.7–5.3% for four of the local models, 4.2–5.0% for the references. Counting a score of 1 or better, everyone lands at about 95%. The local models fall behind only on partial answers, 13–22% against 2.5–5.8%.
What the answers showed
We read the partial and wrong answers behind the numbers. Four patterns account for most of them:
- The asked-for number, without the condition. Four partial answers read in detail each give exactly the value asked for and leave out a secondary key fact: the 370–555 MBq range without the dosimetry note, a 185 MBq activity without the advice to give more without carbidopa, both glucose limits without “reschedule above 200 mg/dL”, and “no precautions” without “after QC and physician approval”. The references think and write at greater length, and include them. The prompt asked for concise answers, which invites the omission.
- Skipped arithmetic. On A001, Qwen3.6 answered 280 MBq in every run, Coder-Next 343 and Gemma 293 once. Both references showed the working and got 327.
- Answering from the wrong passage. On two questions where the right passage wasn’t retrieved, both references said the passages lacked the value. Gemma took a kidney dose for a different tracer family (FAP ligands instead of PSMA) in all three runs, and Coder-Next invented 4.2 mSv for a PSMA tracer. This is the dangerous failure: confident and plausible.
- Retrieval sets the ceiling for everyone. On the 10 questions where the right passage wasn’t in the top 8, every model scored 0–23%. The right move there is to abstain.
The audit found no scorer bugs: every answer parsed, and the partial verdicts we read are what the rubric gives for those answers.
What this means if you build a guideline assistant
- No local model is ready for a staff-facing protocol assistant at this bar. Keep it on a large model for now, hosted or local.
- If you must run locally, use Qwen3.8-27B. It is the only local model with no confident wrong dose, and the most complete.
- Change the prompt. Ask the model to “include any condition or exception attached to the value” and to “show the arithmetic for formula questions”. Both would plausibly close much of the gap, but we didn’t rerun to confirm.
- Gate on retrieval confidence, whichever model you use. Confident answers from the wrong passage are the failure that hurts, and a better generator doesn’t remove it.
- Avoid Nemotron 3.5 here. It answered 27% of the questions it should have declined.
Limits. The questions were written by agents, not department staff. One judge model graded every answer, and it is also one of the references; its agreement with the Opus judge was 96.7% on the answers both graded. The prompt changes above are untested. Retrieval was frozen at one setting (bge-m3, top 8), so a better retriever could raise every model's ceiling.
