Small Models on One DGX Spark

Can a Local Model Screen a Systematic Review? Yes — Here's the Recipe

Ranking diagram: 99 of every 100 included papers land in the top half of the reading list

Screening is the slowest part of a systematic review. A search returns a few thousand papers, and two people read every title and abstract to find the 2–3% that meet the review’s criteria.

A screening assistant doesn’t have to make that call. It only has to rank the papers, so that people who read the top half still see nearly every paper that matters. That’s the question for task L1: can a model running on a desk-side box rank well enough to halve the reading without losing papers?

Small model ready L1 verdict

Yes. Qwen3.8-27B put 99.0% of the human-included papers in the top half of each review, and Gemma 4 put 98.1%. Claude Opus 5.5 scored 100%. Every local model beat a classic active-learning baseline (95.2%).

The few papers missed are ones whose eligibility only shows in the full text, so any abstract-only screener, small or large, would miss them too.

The task in one picture

The model never decides include or exclude. It scores each eligibility criterion separately, and ordinary code turns those scores into a ranking.

InputCriteria + one recordReview's eligibility criteria, split by hand into 4–5 parts, plus one title and abstract
Local modelScore each criterion0–100 per criterion, returned as JSON; 3 samples per record
CodeAverage and sortMean over criteria and samples, then the whole review is ranked
MetricRecall in top 50%What share of the human includes sits in the half people would read?

The data

Four reviews from SYNERGY, an open collection of real systematic-review screening decisions:

Review Records Human includes Topic
Appenzeller-Herzog 2019 582 8 Therapies for Wilson disease
Muthu 2020 923 20 Fragility of spine-surgery RCT results
Ramiro 2015 338 18 Drugs for psoriatic arthritis
Sepriano 2022 1,760 57 DMARD efficacy in rheumatoid arthritis
Total 3,603 103 2.9% of records are includes

The comparison is an ASReview-style active-learning simulation: a TF-IDF and SVM classifier that learns from the reviewer’s decisions as they go, the approach behind common screening tools.

What a call looks like

Every call has the same shape. The abstract itself isn’t reproduced here because of the dataset’s licence terms; titles are bibliographic metadata.

System prompt
You are screening search results for a systematic review, at the
title/abstract stage. For each eligibility criterion, score 0-100 how well
the record meets it: 100 = clearly meets it, 0 = clearly fails it, about
50 = the title/abstract does not say. Judge only from the record.
Return JSON {"scores": [one integer per criterion, in order]}.
User message
Review: Comparative effectiveness of common therapies for Wilson disease

Eligibility criteria:

  1. Population: patients with Wilson disease (WD), any age or stage.
  2. Intervention: D-penicillamine, trientine, tetrathiomolybdate or zinc …
  3. Outcomes: mortality, liver transplantation, neurological or liver symptoms …
  4. Design: prospective or retrospective comparative study …

Record Title: Zinc monotherapy is effective in Wilson’s disease patients with mild liver disease diagnosed in childhood: a retrospective study Abstract: <~2,000 characters>

Gold

Human include: one of the review's 8 includes among 582 records.

Example 1: a clear include, near the top for everyone

Model Scores (run 0) Mean of 3 samples Rank in review
Opus 5.5 [97, 85, 92, 82] 89.0 top 0.2%
DeepSeek-V4.1-Flash [100, 95, 100, 100] 98.8 top 0.2%
Qwen3.8-27B [100, 100, 100, 100] 97.9 top 0.2%
Gemma 4 [100, 100, 100, 100] 100.0 top 0.4%
Qwen3.6 [100, 90, 90, 80] 94.2 top 0.2%
Nemotron 3.5 [100, 0, 100, 50] 60.8 top 0.5%
Qwen3-Coder-Next [100, 75, 100, 100] 96.2 top 0.4%

Nemotron gave the intervention criterion a 0 for a zinc study, which is a plain misreading. The other three criteria still kept the record in its top 0.5%. Scoring criteria separately makes the ranking robust to one bad judgement.

Example 2: an include that abstract screening misses

From Muthu 2020, the record “Posterior lumbar interbody fusion versus posterolateral fusion in spondylolisthesis: a prospective controlled study in the Han nationality”. The criteria require (1) a 1:1 two-arm RCT, (2) spine surgery, (3) a yes/no outcome and (4) a statistically significant result.

Model Scores (run 0) Rank percentile In the top half?
Opus 5.5 [72, 95, 82, 65] 7.6% yes
DeepSeek-V4.1-Flash [85, 100, 85, 45] 18.7% yes
Qwen3.8-27B [100, 100, 50, 0] 26.7% yes
Qwen3.6 [100, 100, 0, 0] 45.7% yes
Qwen3-Coder-Next [50, 100, 0, 0] 56.4% no
Gemma 4 [100, 100, 40, 0] 57.2% no
Nemotron 3.5 [70, 80, 0, 100] 57.2% no

The abstract never shows a yes/no outcome or a significant result, so the local models score criteria 3 and 4 low. The large models give the benefit of the doubt. Neither side is wrong: the paper was included on its full text.

Results

The pass rule was fixed before any model ran: at least 95% recall at 50% reading, and within 2 points of Opus.

Human includes found in the top half of the ranking

Pooled over 4 reviews and 103 includes. One missed paper ≈ 1 point. The axis starts at 90%.

Small modelLarge-model referenceActive learning95% bar
Opus 5.5100.0%
Qwen3.8-27B99.0%
DeepSeek V4.1 Flash98.1%
Gemma 498.1%
Qwen3.697.1%
Nemotron 3.597.1%
Qwen3-Coder-Next95.6%
Active learning95.2%
90%95%100%
Model Recall @50% Avg. precision Work saved @95% Sample agreement (ρ) Verdict
Opus 5.5 100.0% 0.551 0.728 – baseline
DeepSeek-V4.1-Flash 98.1% 0.444 0.612 – reference
Qwen3.8-27B 99.0% 0.415 0.706 0.91 small model ready
Gemma 4 26B-A4B 98.1% 0.257 0.644 0.96 small model ready
Qwen3.6-35B-A3B 97.1% 0.302 0.623 0.88 needs a large model
Nemotron 3.5 30B-A3B 97.1% 0.277 0.633 0.66 needs a large model
Qwen3-Coder-Next 95.6% 0.311 0.671 0.87 needs a large model

With only 103 includes, the line between “small model ready” and “needs a large model” is one or two papers. Qwen3.6 and Nemotron miss three, which puts them 2.9 points behind Opus: just outside the 2-point gap.

61klocal model calls
≈ 14 hGPU time, the largest task in the study
$0.015per 1,000 records locally, vs $6.11 on Opus and $1.01 on DeepSeek

Where the misses come from

  • Muthu 2020 holds most local misses (2–4 per model). Two papers are missed by three or more models, and neither abstract shows what the criteria ask for. Active learning does worst here too, at 76% recall.
  • Sepriano 2022’s denosumab trials are missed by Qwen3.8, Qwen3.6, Nemotron and DeepSeek. Denosumab isn’t a DMARD; the trials were included for structural-damage outcomes. That’s a reasonable borderline call.
  • Nemotron is noisy. Its three samples agree with each other at ρ 0.66, against 0.87–0.96 for the others, so it depends on averaging more than the rest.
  • Ranking precision is lower than Opus’s. Average precision is 0.26–0.42 locally against 0.55, so reading less than half the list is riskier with a local model.

What this means if you run reviews

A local model can do first-pass screening. The recipe:

  1. Split the criteria into separately scorable parts (population, intervention, design, outcome).
  2. Score each criterion 0–100 and let code average them. Don’t ask the model for an include/exclude decision.
  3. Average three samples per record. It raised recall for four of the five models (Gemma 4 was flat).
  4. Read the top half, plus anything a reviewer flags.

Use Qwen3.8-27B for the best recall (it’s the slowest model), or Gemma 4 for speed at 98%. Expect to miss the same papers a large model would: the ones whose eligibility only shows up in the full text.

Limits. Four medical reviews with 1–5% prevalence; other fields may behave differently. The references got one sample each and the local models three, which favours the local models by up to about a point.