Small Models on One DGX Spark

Which Jobs Can a Small Local Model Take Over? Results from Eight Real Tasks

Scorecard of eight tasks: three small-model ready, one needs tuning, four need a large model

Most teams that use a frontier model through an API are paying frontier prices for work that isn’t frontier work: sorting, extracting, filling in forms, following a script. The obvious question is how much of that a small open model could do instead.

The usual answers don’t help much. Leaderboards measure someone else’s tasks, and a model that is “within a few points on average” can still fail in exactly the way that hurts you. So we measured it on eight narrow, realistic tasks, with a pass rule written down for each one before any model ran.

This post is the summary. Each task has its own post with real inputs, real model outputs and the failure analysis; links are at the end.

The short version

Small models reliably handle closed, well-specified work: screening against criteria, extraction into a schema, classification into a fixed label set, and tool-calling loops with a clear policy. On customer-service tool loops, Qwen3.6-35B-A3B matched DeepSeek-V4.1-Flash (90.0% vs 88.7%).

They fall short where the answer must be complete rather than just correct (guideline Q&A), where the work is long and open-ended (repository-level coding: 72.5% vs 95.0%), and where the benchmark encodes conventions the model can't see.

The setup

Five open-weight models, each under 100B parameters and quantised to about 4 bits, were served one at a time by llama.cpp on a single NVIDIA DGX Spark. It’s a desk-side box with a GB10 chip and 121 GB of memory shared by the CPU and GPU.

Model Type Memory Speed (tokens/s)
Qwen3.6-35B-A3B Mixture of experts, 3B active 27.6 GB 66.1
Gemma 4 26B-A4B Mixture of experts, 4B active 27.6 GB 51.5
Nemotron 3.5 30B-A3B Mixture of experts, 3B active 28.2 GB 64.9
Qwen3-Coder-Next Mixture of experts, 80B total, 3B active 55.7 GB 52.2
Qwen3.8-27B Dense, with a speculative draft head 42.4 GB 21.0

“Mixture of experts” means only a small slice of the weights (the “active” parameters) runs for each token, which is why those models are 2.5 to 3 times faster than the dense 27B.

Two large models were the yardstick. Claude Opus 5.5 was the planned frontier baseline. Its API credit ran out on day one, after it had finished six of the eight tasks and 29 tasks of a 30-task subset of a seventh (E4). Those results are kept. DeepSeek-V4.1-Flash, a large open-weight model used through its API, took over as the baseline for the remaining tasks and as the judge where one was needed. It turned out to be a useful second reference in its own right: the realistic cheap alternative. Being open-weight, it can also be run on your own hardware if you have enough of it.

How a task gets its verdict

Each task had two pass bars, fixed in advance. An absolute bar says what’s good enough for the job, such as “finds 95% of the relevant papers”. A relative bar says how far behind the large model is acceptable, such as “within 2 points of Opus”. The verdict follows from which bars the best local model clears.

The verdicts are about model size, not where the model runs. “Needs a large model” doesn’t mean “needs an API”: large open-weight models such as DeepSeek-V4.1-Flash or GLM 5.3 can be deployed locally too, given a multi-GPU server or a cluster rather than one desk-side box.

MeasureBest small modelMedian of 3 sampled runs on the task's primary metric
Both bars metSmall model readyA sub-100B model on one box does the job now
Relative bar onlyNeeds tuningClose to the large model, but under the absolute bar; try fine-tuning
Relative bar missedNeeds a large modelThe gap to the large model is too wide; use a frontier or large open model

The scorecard

Best small model against the large-model references, per task

Primary metric for each task, 0–100 (F1 scores shown ×100). Opus didn't run E1, and its E4 score is a 29-task subset.

Best small modelLarge-model reference (DeepSeek, then Opus)
Small model ready
L1 Screening99.0
98.1
100.0
L2 Structuring100.0
99.9
100.0
E4 Tool loops90.0
88.7
93.1*
Needs tuning
L4 Extraction74.2
77.9
75.8
Needs a large model
E3 Enrichment96.0
97.8
98.1
E2 Log triage86.1
87.5
91.8
L3 Guideline Q&A81.7
90.0
90.8
E1 Coding72.5
95.0
050100
Task What it tests Verdict Best local model Score Baseline
L1 Screening Rank 3,603 abstracts against review criteria small model ready Qwen3.8-27B, Gemma 4 99.0% recall Opus 100%
L2 Report structuring Free-text PET/CT reports → JSON small model ready all five F1 0.98–1.00 Opus 1.000
E4 Tool loops Airline, retail and telecom service agents small model ready Qwen3.6-35B-A3B 90.0% DeepSeek 88.7%
L4 Paper extraction Trial results → numeric cells needs tuning Qwen3.6-35B-A3B 74.2% Opus 75.8%
E3 Enrichment Product offers → 40 categories + fields large model, narrowly Gemma 4 96.0% Opus 98.1%
E2 Log triage Log type, alert severity, incident summary large model, narrowly Qwen3.8-27B F1 0.861 Opus 0.918
L3 Guideline Q&A Answers from 12 clinical guidelines needs a large model Qwen3.8-27B 81.7% Opus 90.8%
E1 Scoped coding 40 real repository bug fixes needs a large model Qwen3.6-35B-A3B 72.5% DeepSeek 95.0%

What each result actually means

Screening and structuring are done. On L1, every local model beat an ASReview-style active-learning baseline, and the best two were within 2 points of Opus. On L2, every model filled the schema almost perfectly. The one thing that separated them was noticing that a report contradicts itself: Opus and DeepSeek caught 100% of the planted contradictions, Qwen3.8 and Gemma 88%, Nemotron 18%. Extraction is easy; noticing the input is inconsistent is harder.

Tool-calling agents are at parity, with guardrails. On E4, Qwen3.6 passed 90.0% of 150 customer-service tasks against DeepSeek’s 88.7%. The two disagree on 22 tasks, 12 one way and 10 the other, so this is a tie, not a win. 130 of its 135 passes needed the right database writes, so it couldn’t pass by just talking. It’s about three times slower per task (130 s against 38 s).

Classification is close but just outside the bar. On E3, Gemma 4 sorted product offers into 40 categories at 96.0%, 2.1 points behind Opus, which is just outside the allowed 2. On E2, alert severity is solved (Gemma 100%, with no critical alert missed). The fault-type gap is mostly log lines the human experts left untagged that read like faults: a labelling convention, not a reading failure.

Guideline answers are rarely wrong, often incomplete. On L3, local models gave a wrong answer about as rarely as the large models (about 5% of answerable questions). They lost points on completeness: they gave the number and dropped the caveat. Only Qwen3.8 met the dose-safety rule of zero confident wrong doses.

Numeric extraction is limited by the answer key. On L4, no model reached the 90% bar, including both large models (75.8% and 77.9%). On 16–19% of cells, both large models agree with each other and not with the gold answer, and most of those look like ambiguous or wrong gold labels.

Real repository coding is where the gap is widest. On E1, DeepSeek fixed 95% of 40 real bugs and the best local model 72.5%. Only two local models solved any of the hardest (1–4 hour) tasks.

How small models fail

Scores tell you whether to deploy. Failure modes tell you which guardrails to build, so they matter as much.

Failure Where we saw it What it looks like Guardrail
Repetition loop Gemma 4 (24% of E4 tasks), Qwen3.8 (8 E1 tasks) The same tool call with the same arguments, 71 times in one case Stop or re-prompt after k identical calls
Invented identifiers Nemotron (55% of E4 tasks loop or error) Calls tools with made-up ids like user john_doe_123 instead of asking Reject arguments not seen in the conversation or a tool result
Burning the step budget Qwen3-Coder-Next (60% of E1 runs hit the cap) Writes a correct fix, then keeps re-running tests An explicit stop rule; a budget per phase
Incomplete answers Every local model (L3) The right number with a missing caveat Ask for required elements; checklist grading
Missed contradictions Nemotron 18%, Coder-Next 65% caught (L2) Picks one of two conflicting values without a flag A separate consistency check
Over-filling empty fields Fine-tuned Qwen3.6 (L4) Gives a value where the answer is “not reported” Train on realistic empty cases

Does fine-tuning close the gap?

L4 was the one task that landed on “needs tuning”, so Qwen3.6 got a LoRA (a small set of extra trainable weights, 8.4M parameters here) trained on 1,232 synthetic trial papers. It learned the synthetic task: 90.0% → 94.6% on held-out synthetic papers. On the real papers it went from 74.2% to 73.8%.

It got better at reading reported values (+7 to +9 points on group sizes) but also learned to answer more often, while 28% of the real answer key is “not reported” by annotator convention. The two effects cancelled out. The lesson: a LoRA learns what its training data contains. Synthetic data has to copy the hard cases of the real data, not just its format.

Practical guidelines

Small-model ready

  • Schema-bound extraction from clean text (L2): any of the five models, with JSON-schema constrained decoding. Add a separate consistency check if inputs can contradict themselves.
  • Screening against explicit criteria (L1): high recall, cheap enough to run on every record. Keep a person on the includes.
  • Classification into a fixed label set (E2 severity, E3 categories): 96–100%, with no invented labels. Gemma 4 led both.
  • Policy-driven tool loops with a bounded set of actions (E4): Qwen3.6-35B-A3B, with a repeated-call guard and an id check around it.

Usable with review

  • Label conventions the model can’t see (E2, E3): within 2–6 points of the frontier. Show examples of the convention, or ask a large model for a second opinion on low-confidence items.
  • Numbers from scientific papers (L4): about 74% of cells, close to the large models. Fine for a first draft a person checks; not for unattended meta-analysis.
  • Retrieval-augmented answers (L3): rarely wrong and good at saying “not in the guideline”, but often incomplete. Fine for lookup, not for safety-relevant answers such as doses.

Needs a large model (frontier API or large open model)

  • Open-ended coding in a real repository (E1): a 22.5-point gap even with five times the step budget.
  • Safety-critical answers that must be complete (L3 dose questions).
  • Anywhere silent inconsistency or invented ids would be costly, unless the guardrails above are in place.

Which model for what

Model Strengths Weaknesses Use it for
Qwen3.6-35B-A3B Best agent (E4, E1); fast; 28 GB Weaker on L3 completeness The default local model; tool loops
Qwen3.8-27B Most accurate on judgement and Q&A (L1, L3, E2) Slow (21 tokens/s); loops in E1 Batch screening and Q&A, not interactive agents
Gemma 4 26B-A4B Classification and summaries Repetition loops as an agent Classification; never unguarded tool loops
Qwen3-Coder-Next Decent coding with a big budget 56 GB; burns steps Not preferred over Qwen3.6
Nemotron 3.5 30B-A3B Fast decode Invents ids; weak abstention and consistency Not recommended for these tasks

What about cost?

$0.003–3.60per 1,000 examples locally, across the eight tasks
$0.26–124per 1,000 on DeepSeek-V4.1-Flash at list price
$0.87–41per 1,000 on Opus 5.5 at list price (six full tasks)

Local cost here uses the study’s amortised €0.03 per GPU-hour. On the small-model-ready tasks, local inference is two to three orders of magnitude cheaper than Opus, and roughly 10 to 80 times cheaper than DeepSeek. Cost rarely decides it, though: accuracy and failure modes do. A cheap model that loops on a quarter of your agent tasks isn’t cheap.

Limits. Opus is missing on E1 and partial on E4, so those verdicts use DeepSeek as the baseline. E4 had one trial per task (about ±2.5 points). E1 is cut down to 40 django and sympy tasks that run on arm64. L2 and E2 severity use synthetic data and are at ceiling, and L3's questions were written by agents and checked separately, not by clinicians. Every local model ran at about 4 bits on llama.cpp; higher precision or another runtime could score differently. Only one model was fine-tuned, on one task.

How the whole study fitted on one box in eight days, and what went wrong along the way, is part 2.