Series · 12 parts
Small Models on One DGX Spark
Five open-weight models, eight real tasks, one desk-side box. A measured answer to the question every team asks: which of our large-model calls could a small local model take over?
12 parts · November 2026 · 67AI Lab
Every team using a frontier model eventually asks the same thing: how much of
this could a small open model do on our own hardware? Benchmarks don’t
answer it, because leaderboard tasks aren’t your tasks, and a model that’s
“close on average” can fail in exactly the way that hurts you.
So we ran the experiment. We took five open-weight models under 100B
parameters, served them one at a time on a single
NVIDIA DGX Spark
(GB10, 121 GB unified memory), and gave them eight narrow, realistic
tasks. Each task had a pass rule written down before any model ran. Every
result was compared with Claude Opus 5.5 and DeepSeek-V4.1-Flash, and
every surprising score was checked by reading the traces.
3tasks small-model ready: screening, report structuring, tool-calling agents
1task that needs tuning: numeric extraction from trial papers
4tasks that still need a large model, two of them only narrowly
≈ 62 hof GPU time on one box, over 8 days
The verdict on one page
Best small model against the best large-model reference, per task
Primary metric for each task, 0–100 (F1 scores shown ×100). Status: whether the best local model met both pass bars.
Best small modelLarge-model reference (Opus; DeepSeek for E1 and E4)
Small model ready
L1 Screening99.0
100.0
L2 Structuring100.0
100.0
E4 Tool loops90.0
88.7
Needs tuning
L4 Extraction74.2
77.9
Needs a large model
E3 Enrichment96.0
98.1
E2 Log triage86.1
91.8
L3 Guideline Q&A81.7
90.8
E1 Coding72.5
95.0
050100
The pattern in one sentence: small models are ready for closed,
well-specified work, such as judging one item against explicit criteria,
filling a schema or following a tool policy. They fall short where the
answer has to be complete, the work is long and open-ended, or the
benchmark encodes conventions the model can’t see.
The three verdicts are about model size, not where the model runs.
“Needs a large model” doesn’t mean “needs an API”: large open-weight models
such as DeepSeek-V4.1-Flash or GLM 5.3 can be deployed locally too, given
more hardware than one desk-side box.
How to read this series
Start with part 1 for the results and the practical guidelines, and
part 2 if you want to run something similar yourself. Each task post
(parts 3–12) stands on its own and follows the same shape: what the task is
and why it matters, a real input with the real model outputs, the results
against the pass bar, what the traces showed, and what it means if you do
this work.
II.Getting the models running
Before any task: can each model be served fast enough, without silently truncating prompts or breaking tool calls?
-
03
Before Any Benchmark: Can Five Open Models Run Well on One DGX Spark?T01: all five models kept a 32k-token prompt intact and finished a 5-call tool chain. The four MoE models decode at 52–66 tok/s; the dense Qwen3.8-27B manages 21.
9 min
III.Life-science and clinical tasks
Reading medical literature, structuring scan reports, answering from guidelines and extracting trial numbers.
-
04
Can a Local Model Screen a Systematic Review? Yes — Here's the RecipeL1: five open models ranked 3,603 abstracts from four medical systematic reviews. The best put 99% of the papers humans included in the top half, within a point of Claude Opus 5.5.
7 min
-
05
Turning Scan Reports into JSON Is Solved. Spotting Contradictions Isn'tL2: every local model turned 300 synthetic PET/CT and SPECT reports into valid JSON at field F1 ≥ 0.98. Catching planted contradictions ranged from 88% (Qwen3.8, Gemma 4) to 18% (Nemotron).
8 min
-
06
Rarely Wrong, Often Incomplete: Small Models Answering from Clinical GuidelinesL3: the best local model, Qwen3.8-27B, fully answered 81.7% of answerable guideline questions against 90.8% for Claude Opus 5.5. Local models are wrong as rarely as the references; they drop the caveats.
9 min
-
07
When the Answer Key Is the Ceiling: Pulling Numbers out of Trial PapersL4: no model, small or large, reached 90% on extracting trial results for meta-analysis. Qwen3.6-35B-A3B scored 74.2%, 1.6 points behind Claude Opus 5.5, and the answer key itself caps the score.
9 min
IV.Engineering and enterprise tasks
Fixing real bugs in real repositories, triaging logs, cleaning product data and running customer-service agents.
-
08
Fixing Real Bugs in Real Repos: Where Small Models Still Trail by 22.5 PointsE1: five local models drove a coding agent on 40 real SWE-bench issues. The best, Qwen3.6-35B-A3B, fixed 72.5%; DeepSeek-V4.1-Flash fixed 95%. The widest small-vs-large gap in the study.
10 min
-
09
Log Triage: Severity Is Solved, Fault Labels Are a ConventionE2: Qwen3.8-27B classifies log lines at macro-F1 0.861, 5.7 points behind Claude Opus 5.5. Three local models rate alert severity at 98.8–100%, and the gap is mostly lines the experts left untagged.
9 min
-
10
2,360 Offers, 40 Categories: Small Models Land Two Points Short on Product DataE3: Gemma 4 filed 96.0% of 2,360 web-shop offers in the right category, for under a cent per thousand. Claude Opus 5.5 scored 98.1%, so the rule says it still needs a large model, by 0.08 points.
8 min
-
11
A 3B-Active Model That Matches DeepSeek at Running Customer-Service AgentsE4: on 150 simulated customer-service tasks, Qwen3.6-35B-A3B passed 90.0% against 88.7% for DeepSeek-V4.1-Flash. The strongest local result in the study; two other models looped or invented ids.
9 min
V.Can fine-tuning close the gap?
A LoRA on the one task that landed on “needs tuning”, and why it learned the wrong lesson.
-
12
The LoRA That Learned the Wrong Lesson: Fine-Tuning on Synthetic PapersPhase 4: a LoRA on Qwen3.6-35B-A3B lifted synthetic-paper extraction from 90.0% to 94.6%, but real L4 went from 74.2% to 73.8%. It learned to answer more often, including where the answer key is empty.
9 min