Can a Local Model Screen a Systematic Review? Yes — Here's the Recipe
Screening is the slowest part of a systematic review. A search returns a few thousand papers, and two people read every title and abstract to find the 2–3% that meet the review’s criteria.
A screening assistant doesn’t have to make that call. It only has to rank the papers, so that people who read the top half still see nearly every paper that matters. That’s the question for task L1: can a model running on a desk-side box rank well enough to halve the reading without losing papers?
Yes. Qwen3.8-27B put 99.0% of the human-included papers in the top half of each review, and Gemma 4 put 98.1%. Claude Opus 5.5 scored 100%. Every local model beat a classic active-learning baseline (95.2%).
The few papers missed are ones whose eligibility only shows in the full text, so any abstract-only screener, small or large, would miss them too.
The task in one picture
The model never decides include or exclude. It scores each eligibility criterion separately, and ordinary code turns those scores into a ranking.
The data
Four reviews from SYNERGY, an open collection of real systematic-review screening decisions:
| Review | Records | Human includes | Topic |
|---|---|---|---|
| Appenzeller-Herzog 2019 | 582 | 8 | Therapies for Wilson disease |
| Muthu 2020 | 923 | 20 | Fragility of spine-surgery RCT results |
| Ramiro 2015 | 338 | 18 | Drugs for psoriatic arthritis |
| Sepriano 2022 | 1,760 | 57 | DMARD efficacy in rheumatoid arthritis |
| Total | 3,603 | 103 | 2.9% of records are includes |
The comparison is an ASReview-style active-learning simulation: a TF-IDF and SVM classifier that learns from the reviewer’s decisions as they go, the approach behind common screening tools.
What a call looks like
Every call has the same shape. The abstract itself isn’t reproduced here because of the dataset’s licence terms; titles are bibliographic metadata.
You are screening search results for a systematic review, at the
title/abstract stage. For each eligibility criterion, score 0-100 how well
the record meets it: 100 = clearly meets it, 0 = clearly fails it, about
50 = the title/abstract does not say. Judge only from the record.
Return JSON {"scores": [one integer per criterion, in order]}.Review: Comparative effectiveness of common therapies for Wilson diseaseEligibility criteria:
- Population: patients with Wilson disease (WD), any age or stage.
- Intervention: D-penicillamine, trientine, tetrathiomolybdate or zinc …
- Outcomes: mortality, liver transplantation, neurological or liver symptoms …
- Design: prospective or retrospective comparative study …
Record Title: Zinc monotherapy is effective in Wilson’s disease patients with mild liver disease diagnosed in childhood: a retrospective study Abstract: <~2,000 characters>
Human include: one of the review's 8 includes among 582 records.
Example 1: a clear include, near the top for everyone
| Model | Scores (run 0) | Mean of 3 samples | Rank in review |
|---|---|---|---|
| Opus 5.5 | [97, 85, 92, 82] | 89.0 | top 0.2% |
| DeepSeek-V4.1-Flash | [100, 95, 100, 100] | 98.8 | top 0.2% |
| Qwen3.8-27B | [100, 100, 100, 100] | 97.9 | top 0.2% |
| Gemma 4 | [100, 100, 100, 100] | 100.0 | top 0.4% |
| Qwen3.6 | [100, 90, 90, 80] | 94.2 | top 0.2% |
| Nemotron 3.5 | [100, 0, 100, 50] | 60.8 | top 0.5% |
| Qwen3-Coder-Next | [100, 75, 100, 100] | 96.2 | top 0.4% |
Nemotron gave the intervention criterion a 0 for a zinc study, which is a plain misreading. The other three criteria still kept the record in its top 0.5%. Scoring criteria separately makes the ranking robust to one bad judgement.
Example 2: an include that abstract screening misses
From Muthu 2020, the record “Posterior lumbar interbody fusion versus posterolateral fusion in spondylolisthesis: a prospective controlled study in the Han nationality”. The criteria require (1) a 1:1 two-arm RCT, (2) spine surgery, (3) a yes/no outcome and (4) a statistically significant result.
| Model | Scores (run 0) | Rank percentile | In the top half? |
|---|---|---|---|
| Opus 5.5 | [72, 95, 82, 65] | 7.6% | yes |
| DeepSeek-V4.1-Flash | [85, 100, 85, 45] | 18.7% | yes |
| Qwen3.8-27B | [100, 100, 50, 0] | 26.7% | yes |
| Qwen3.6 | [100, 100, 0, 0] | 45.7% | yes |
| Qwen3-Coder-Next | [50, 100, 0, 0] | 56.4% | no |
| Gemma 4 | [100, 100, 40, 0] | 57.2% | no |
| Nemotron 3.5 | [70, 80, 0, 100] | 57.2% | no |
The abstract never shows a yes/no outcome or a significant result, so the local models score criteria 3 and 4 low. The large models give the benefit of the doubt. Neither side is wrong: the paper was included on its full text.
Results
The pass rule was fixed before any model ran: at least 95% recall at 50% reading, and within 2 points of Opus.
Human includes found in the top half of the ranking
Pooled over 4 reviews and 103 includes. One missed paper ≈ 1 point. The axis starts at 90%.
| Model | Recall @50% | Avg. precision | Work saved @95% | Sample agreement (ρ) | Verdict |
|---|---|---|---|---|---|
| Opus 5.5 | 100.0% | 0.551 | 0.728 | – | baseline |
| DeepSeek-V4.1-Flash | 98.1% | 0.444 | 0.612 | – | reference |
| Qwen3.8-27B | 99.0% | 0.415 | 0.706 | 0.91 | small model ready |
| Gemma 4 26B-A4B | 98.1% | 0.257 | 0.644 | 0.96 | small model ready |
| Qwen3.6-35B-A3B | 97.1% | 0.302 | 0.623 | 0.88 | needs a large model |
| Nemotron 3.5 30B-A3B | 97.1% | 0.277 | 0.633 | 0.66 | needs a large model |
| Qwen3-Coder-Next | 95.6% | 0.311 | 0.671 | 0.87 | needs a large model |
With only 103 includes, the line between “small model ready” and “needs a large model” is one or two papers. Qwen3.6 and Nemotron miss three, which puts them 2.9 points behind Opus: just outside the 2-point gap.
Where the misses come from
- Muthu 2020 holds most local misses (2–4 per model). Two papers are missed by three or more models, and neither abstract shows what the criteria ask for. Active learning does worst here too, at 76% recall.
- Sepriano 2022’s denosumab trials are missed by Qwen3.8, Qwen3.6, Nemotron and DeepSeek. Denosumab isn’t a DMARD; the trials were included for structural-damage outcomes. That’s a reasonable borderline call.
- Nemotron is noisy. Its three samples agree with each other at ρ 0.66, against 0.87–0.96 for the others, so it depends on averaging more than the rest.
- Ranking precision is lower than Opus’s. Average precision is 0.26–0.42 locally against 0.55, so reading less than half the list is riskier with a local model.
What this means if you run reviews
A local model can do first-pass screening. The recipe:
- Split the criteria into separately scorable parts (population, intervention, design, outcome).
- Score each criterion 0–100 and let code average them. Don’t ask the model for an include/exclude decision.
- Average three samples per record. It raised recall for four of the five models (Gemma 4 was flat).
- Read the top half, plus anything a reviewer flags.
Use Qwen3.8-27B for the best recall (it’s the slowest model), or Gemma 4 for speed at 98%. Expect to miss the same papers a large model would: the ones whose eligibility only shows up in the full text.
Limits. Four medical reviews with 1–5% prevalence; other fields may behave differently. The references got one sample each and the local models three, which favours the local models by up to about a point.
