Small Models on One DGX Spark

Rarely Wrong, Often Incomplete: Small Models Answering from Clinical Guidelines

Qwen3.8-27B 9 points behind on fully correct guideline answers: 90.8% Opus, 90.0% DeepSeek, 81.7% Qwen3.8-27B against an 85% bar

Technologists and physicians in a nuclear-medicine department look up dosing, patient preparation and safety rules in procedure guidelines many times a day. An assistant that answers from those guidelines would save time, and running it locally would keep every question inside the hospital.

The bar is strict, though. An answer must carry every key fact, because the caveat attached to a number is often the safety part. The assistant must say so when the guideline doesn’t cover a question. And a confident wrong dose rules a model out. Task L3 asks whether any local model clears that bar.

Needs a large model L3 verdict

Not yet. The best local model, Qwen3.8-27B, fully answered 81.7% of the answerable questions, against 90.8% for Claude Opus 5.5. The bar was 85% and within 5 points of Opus.

The gap isn't wrong answers. Local models are wrong about as rarely as the references, around 5% of answers. They lose on completeness: they give the number that was asked for and drop the condition attached to it. Only Qwen3.8-27B never gave a confident wrong dose.

The task in one picture

This is retrieval-augmented generation (RAG): a search step finds the relevant guideline passages, and the model answers from those passages only. We froze the search results, so the model is the only thing that changes between runs.

InputA staff questionAbout dosing, preparation or safety, from 12 procedure guidelines
RetrievalTop 8 passagesFound once with the bge-m3 embedding model, then frozen
Local modelAnswer or abstainJSON with the answer, whether it's answerable, and the passages cited
JudgeBlind gradingDeepSeek-V4.1-Flash scores 0–2 against the reference answer and key facts

The data

The corpus is 12 open-access procedure guidelines from the EANM and SNMMI, the European and North American nuclear-medicine societies, split into passages of about 512 tokens. The guideline texts aren’t reproduced here or committed to the study’s repository.

What Detail
Questions 150: 120 answerable, 30 unanswerable (10 and 2–3 per guideline)
Dose questions 70 of the 150
Each question has a reference answer, the key facts it must contain, grading notes
Written by three authoring agents, then checked in a separate pass: 123 ok, 21 edited, 6 replaced
Retrieval the right passage is in the top 8 for 110 of the 120 answerable questions

The study plan asked for questions written by department staff; agent-written questions are a deviation. The unanswerable questions are ones the guideline leaves to national rules or doesn’t address, and the right reply is to say so.

How answers are graded. A blind judge, DeepSeek-V4.1-Flash, sees the question, the reference answer, the key facts, the grading notes and the answer with the model’s name removed. It scores 2 when every key fact is right, 1 for partly right, and 0 for wrong. Claude Opus 5.5 also graded Opus’s own 150 answers before its API credit ran out, and the two judges gave the same verdict on 96.7% of them.

What a call looks like

Each call carries the question and the eight frozen passages. The system prompt tells the model to answer only from the passages, give exact numbers and units, and never guess a dose. Two examples show the two ways answers go wrong. The passages themselves are guideline text and aren’t shown.

Question A001 (dose question)

We image at 3 min per bed position on a scanner with 30% or less bed overlap. Using the EANM linear formula, what's the minimum FDG activity for a 70 kg adult?

Reference answer
FDG (MBq) = 14 MBq·min·bed⁻¹·kg⁻¹ × weight (kg) / min per bed
          = 14 × 70 / 3 ≈ 327 MBq minimum
Opus 5.5 and Qwen3.8-27B score 2

Both write out the working, 14 × 70 / 3 = 980 / 3 ≈ 326.7 MBq, and answer about 327 MBq.

Four local models score 0

Gemma 4: 293.33 MBq. Qwen3-Coder-Next: 343 MBq. Qwen3.6: 280 MBq. Nemotron 3.5: 1470 MBq.

Every model cited the right passage, and the formula and inputs are both in it. The four wrong answers skipped the arithmetic and stated a number anyway. These count as confident wrong dose answers, the failure the safety rule exists to catch.

The second example is the more common failure. Question B021 asks which range of fixed iodine-131 activities is most often used to treat Graves’ disease. The key facts are the range, 370–555 MBq, and the note that a dosimetric approach should be considered for thyroid volumes above 40 ml.

Model Answer (run 0) Score
DeepSeek-V4.1-Flash The 370–555 MBq range, plus the dosimetry note for glands above 40 ml 2
Qwen3.8-27B, Qwen3.6 “370 to 555 MBq” 1
Gemma 4 The same range, in one sentence, without the dosimetry note 1

The local answers are correct and incomplete. A reader who only needed the number is fine; a reader with a large gland misses the part that matters.

The unanswerable questions went well. One asks for the dose-rate limit a patient must be below before discharge after lutetium-177 PSMA therapy, which the guideline leaves to national legislation. All seven models said the passages don’t give a number.

Results

The pass rule, fixed before any model ran: at least 85% fully correct, at least 90% correct abstention on unanswerable questions, zero confident wrong dose answers in any run, and within 5 points of Opus.

Answerable questions answered fully correctly (score 2)

120 answerable questions; median of 3 runs for local models. Scale 0–100%.

Small modelLarge-model reference85% bar
Opus 5.590.8%
DeepSeek V4.1 Flash90.0%
Qwen3.8-27B81.7%
Qwen3-Coder-Next80.0%
Gemma 479.2%
Qwen3.675.8%
Nemotron 3.571.7%
0%25%50%75%100%
Model Fully correct Correct when right passage retrieved Correct abstention Wrong doses, worst run Δ vs Opus Verdict
Opus 5.5 90.8% 98.2% 100% 0 – baseline
DeepSeek-V4.1-Flash 90.0% 96.4% 100% 0 −0.8 reference
Qwen3.8-27B 81.7% 88.2% 100% 0 −9.2 needs a large model
Qwen3-Coder-Next 80.0% 85.5% 96.7% 3 −10.8 needs a large model
Gemma 4 26B-A4B 79.2% 86.4% 100% 2 −11.7 needs a large model
Qwen3.6-35B-A3B 75.8% 81.8% 100% 1 −15.0 needs a large model
Nemotron 3.5 30B-A3B 71.7% 76.4% 73.3% 12 −19.2 needs a large model

Abstention is the one part every model but Nemotron passes. Nemotron asserted an answer on 27% of the unanswerable questions and gave 12 confident wrong doses in its worst run.

Wrong as rarely, but less complete

Splitting the scores into wrong, partial and correct shows where the gap is.

How the answers fail

Share of the 120 answerable questions, pooled over the sampled runs (references: 1 run). The axis runs from 0 to 25%.

Small modelLarge-model reference
Wrong (score 0)
Opus 5.55.0%
DeepSeek V4.1 Flash4.2%
Qwen3.8-27B5.3%
Qwen3-Coder-Next4.7%
Gemma 45.3%
Qwen3.64.7%
Nemotron 3.56.9%
Partial (score 1)
Opus 5.52.5%
DeepSeek V4.1 Flash5.8%
Qwen3.8-27B13.1%
Qwen3-Coder-Next15.0%
Gemma 415.8%
Qwen3.619.7%
Nemotron 3.522.2%
0%5%10%15%20%25%

The wrong-answer rate is flat: 4.7–5.3% for four of the local models, 4.2–5.0% for the references. Counting a score of 1 or better, everyone lands at about 95%. The local models fall behind only on partial answers, 13–22% against 2.5–5.8%.

≈ 3.2 hGPU time for 3.3k local calls, plus 3.5k judge calls
110 / 120answerable questions with the right passage in the top 8
96.7%agreement between the DeepSeek and Opus judges
$0.024–0.064per 1,000 questions locally, vs $27.65 on Opus and $1.27 on DeepSeek

What the answers showed

We read the partial and wrong answers behind the numbers. Four patterns account for most of them:

  • The asked-for number, without the condition. Four partial answers read in detail each give exactly the value asked for and leave out a secondary key fact: the 370–555 MBq range without the dosimetry note, a 185 MBq activity without the advice to give more without carbidopa, both glucose limits without “reschedule above 200 mg/dL”, and “no precautions” without “after QC and physician approval”. The references think and write at greater length, and include them. The prompt asked for concise answers, which invites the omission.
  • Skipped arithmetic. On A001, Qwen3.6 answered 280 MBq in every run, Coder-Next 343 and Gemma 293 once. Both references showed the working and got 327.
  • Answering from the wrong passage. On two questions where the right passage wasn’t retrieved, both references said the passages lacked the value. Gemma took a kidney dose for a different tracer family (FAP ligands instead of PSMA) in all three runs, and Coder-Next invented 4.2 mSv for a PSMA tracer. This is the dangerous failure: confident and plausible.
  • Retrieval sets the ceiling for everyone. On the 10 questions where the right passage wasn’t in the top 8, every model scored 0–23%. The right move there is to abstain.

The audit found no scorer bugs: every answer parsed, and the partial verdicts we read are what the rubric gives for those answers.

What this means if you build a guideline assistant

  1. No local model is ready for a staff-facing protocol assistant at this bar. Keep it on a large model for now, hosted or local.
  2. If you must run locally, use Qwen3.8-27B. It is the only local model with no confident wrong dose, and the most complete.
  3. Change the prompt. Ask the model to “include any condition or exception attached to the value” and to “show the arithmetic for formula questions”. Both would plausibly close much of the gap, but we didn’t rerun to confirm.
  4. Gate on retrieval confidence, whichever model you use. Confident answers from the wrong passage are the failure that hurts, and a better generator doesn’t remove it.
  5. Avoid Nemotron 3.5 here. It answered 27% of the questions it should have declined.

Limits. The questions were written by agents, not department staff. One judge model graded every answer, and it is also one of the references; its agreement with the Opus judge was 96.7% on the answers both graded. The prompt changes above are untested. Retrieval was frozen at one setting (bge-m3, top 8), so a better retriever could raise every model's ceiling.