Which Jobs Can a Small Local Model Take Over? Results from Eight Real Tasks
Most teams that use a frontier model through an API are paying frontier prices for work that isn’t frontier work: sorting, extracting, filling in forms, following a script. The obvious question is how much of that a small open model could do instead.
The usual answers don’t help much. Leaderboards measure someone else’s tasks, and a model that is “within a few points on average” can still fail in exactly the way that hurts you. So we measured it on eight narrow, realistic tasks, with a pass rule written down for each one before any model ran.
This post is the summary. Each task has its own post with real inputs, real model outputs and the failure analysis; links are at the end.
Small models reliably handle closed, well-specified work: screening against criteria, extraction into a schema, classification into a fixed label set, and tool-calling loops with a clear policy. On customer-service tool loops, Qwen3.6-35B-A3B matched DeepSeek-V4.1-Flash (90.0% vs 88.7%).
They fall short where the answer must be complete rather than just correct (guideline Q&A), where the work is long and open-ended (repository-level coding: 72.5% vs 95.0%), and where the benchmark encodes conventions the model can't see.
The setup
Five open-weight models, each under 100B parameters and quantised to about 4 bits, were served one at a time by llama.cpp on a single NVIDIA DGX Spark. It’s a desk-side box with a GB10 chip and 121 GB of memory shared by the CPU and GPU.
| Model | Type | Memory | Speed (tokens/s) |
|---|---|---|---|
| Qwen3.6-35B-A3B | Mixture of experts, 3B active | 27.6 GB | 66.1 |
| Gemma 4 26B-A4B | Mixture of experts, 4B active | 27.6 GB | 51.5 |
| Nemotron 3.5 30B-A3B | Mixture of experts, 3B active | 28.2 GB | 64.9 |
| Qwen3-Coder-Next | Mixture of experts, 80B total, 3B active | 55.7 GB | 52.2 |
| Qwen3.8-27B | Dense, with a speculative draft head | 42.4 GB | 21.0 |
“Mixture of experts” means only a small slice of the weights (the “active” parameters) runs for each token, which is why those models are 2.5 to 3 times faster than the dense 27B.
Two large models were the yardstick. Claude Opus 5.5 was the planned frontier baseline. Its API credit ran out on day one, after it had finished six of the eight tasks and 29 tasks of a 30-task subset of a seventh (E4). Those results are kept. DeepSeek-V4.1-Flash, a large open-weight model used through its API, took over as the baseline for the remaining tasks and as the judge where one was needed. It turned out to be a useful second reference in its own right: the realistic cheap alternative. Being open-weight, it can also be run on your own hardware if you have enough of it.
How a task gets its verdict
Each task had two pass bars, fixed in advance. An absolute bar says what’s good enough for the job, such as “finds 95% of the relevant papers”. A relative bar says how far behind the large model is acceptable, such as “within 2 points of Opus”. The verdict follows from which bars the best local model clears.
The verdicts are about model size, not where the model runs. “Needs a large model” doesn’t mean “needs an API”: large open-weight models such as DeepSeek-V4.1-Flash or GLM 5.3 can be deployed locally too, given a multi-GPU server or a cluster rather than one desk-side box.
The scorecard
Best small model against the large-model references, per task
Primary metric for each task, 0–100 (F1 scores shown ×100). Opus didn't run E1, and its E4 score is a 29-task subset.
| Task | What it tests | Verdict | Best local model | Score | Baseline |
|---|---|---|---|---|---|
| L1 Screening | Rank 3,603 abstracts against review criteria | small model ready | Qwen3.8-27B, Gemma 4 | 99.0% recall | Opus 100% |
| L2 Report structuring | Free-text PET/CT reports → JSON | small model ready | all five | F1 0.98–1.00 | Opus 1.000 |
| E4 Tool loops | Airline, retail and telecom service agents | small model ready | Qwen3.6-35B-A3B | 90.0% | DeepSeek 88.7% |
| L4 Paper extraction | Trial results → numeric cells | needs tuning | Qwen3.6-35B-A3B | 74.2% | Opus 75.8% |
| E3 Enrichment | Product offers → 40 categories + fields | large model, narrowly | Gemma 4 | 96.0% | Opus 98.1% |
| E2 Log triage | Log type, alert severity, incident summary | large model, narrowly | Qwen3.8-27B | F1 0.861 | Opus 0.918 |
| L3 Guideline Q&A | Answers from 12 clinical guidelines | needs a large model | Qwen3.8-27B | 81.7% | Opus 90.8% |
| E1 Scoped coding | 40 real repository bug fixes | needs a large model | Qwen3.6-35B-A3B | 72.5% | DeepSeek 95.0% |
What each result actually means
Screening and structuring are done. On L1, every local model beat an ASReview-style active-learning baseline, and the best two were within 2 points of Opus. On L2, every model filled the schema almost perfectly. The one thing that separated them was noticing that a report contradicts itself: Opus and DeepSeek caught 100% of the planted contradictions, Qwen3.8 and Gemma 88%, Nemotron 18%. Extraction is easy; noticing the input is inconsistent is harder.
Tool-calling agents are at parity, with guardrails. On E4, Qwen3.6 passed 90.0% of 150 customer-service tasks against DeepSeek’s 88.7%. The two disagree on 22 tasks, 12 one way and 10 the other, so this is a tie, not a win. 130 of its 135 passes needed the right database writes, so it couldn’t pass by just talking. It’s about three times slower per task (130 s against 38 s).
Classification is close but just outside the bar. On E3, Gemma 4 sorted product offers into 40 categories at 96.0%, 2.1 points behind Opus, which is just outside the allowed 2. On E2, alert severity is solved (Gemma 100%, with no critical alert missed). The fault-type gap is mostly log lines the human experts left untagged that read like faults: a labelling convention, not a reading failure.
Guideline answers are rarely wrong, often incomplete. On L3, local models gave a wrong answer about as rarely as the large models (about 5% of answerable questions). They lost points on completeness: they gave the number and dropped the caveat. Only Qwen3.8 met the dose-safety rule of zero confident wrong doses.
Numeric extraction is limited by the answer key. On L4, no model reached the 90% bar, including both large models (75.8% and 77.9%). On 16–19% of cells, both large models agree with each other and not with the gold answer, and most of those look like ambiguous or wrong gold labels.
Real repository coding is where the gap is widest. On E1, DeepSeek fixed 95% of 40 real bugs and the best local model 72.5%. Only two local models solved any of the hardest (1–4 hour) tasks.
How small models fail
Scores tell you whether to deploy. Failure modes tell you which guardrails to build, so they matter as much.
| Failure | Where we saw it | What it looks like | Guardrail |
|---|---|---|---|
| Repetition loop | Gemma 4 (24% of E4 tasks), Qwen3.8 (8 E1 tasks) | The same tool call with the same arguments, 71 times in one case | Stop or re-prompt after k identical calls |
| Invented identifiers | Nemotron (55% of E4 tasks loop or error) | Calls tools with made-up ids like user john_doe_123 instead of asking |
Reject arguments not seen in the conversation or a tool result |
| Burning the step budget | Qwen3-Coder-Next (60% of E1 runs hit the cap) | Writes a correct fix, then keeps re-running tests | An explicit stop rule; a budget per phase |
| Incomplete answers | Every local model (L3) | The right number with a missing caveat | Ask for required elements; checklist grading |
| Missed contradictions | Nemotron 18%, Coder-Next 65% caught (L2) | Picks one of two conflicting values without a flag | A separate consistency check |
| Over-filling empty fields | Fine-tuned Qwen3.6 (L4) | Gives a value where the answer is “not reported” | Train on realistic empty cases |
Does fine-tuning close the gap?
L4 was the one task that landed on “needs tuning”, so Qwen3.6 got a LoRA (a small set of extra trainable weights, 8.4M parameters here) trained on 1,232 synthetic trial papers. It learned the synthetic task: 90.0% → 94.6% on held-out synthetic papers. On the real papers it went from 74.2% to 73.8%.
It got better at reading reported values (+7 to +9 points on group sizes) but also learned to answer more often, while 28% of the real answer key is “not reported” by annotator convention. The two effects cancelled out. The lesson: a LoRA learns what its training data contains. Synthetic data has to copy the hard cases of the real data, not just its format.
Practical guidelines
Small-model ready
- Schema-bound extraction from clean text (L2): any of the five models, with JSON-schema constrained decoding. Add a separate consistency check if inputs can contradict themselves.
- Screening against explicit criteria (L1): high recall, cheap enough to run on every record. Keep a person on the includes.
- Classification into a fixed label set (E2 severity, E3 categories): 96–100%, with no invented labels. Gemma 4 led both.
- Policy-driven tool loops with a bounded set of actions (E4): Qwen3.6-35B-A3B, with a repeated-call guard and an id check around it.
Usable with review
- Label conventions the model can’t see (E2, E3): within 2–6 points of the frontier. Show examples of the convention, or ask a large model for a second opinion on low-confidence items.
- Numbers from scientific papers (L4): about 74% of cells, close to the large models. Fine for a first draft a person checks; not for unattended meta-analysis.
- Retrieval-augmented answers (L3): rarely wrong and good at saying “not in the guideline”, but often incomplete. Fine for lookup, not for safety-relevant answers such as doses.
Needs a large model (frontier API or large open model)
- Open-ended coding in a real repository (E1): a 22.5-point gap even with five times the step budget.
- Safety-critical answers that must be complete (L3 dose questions).
- Anywhere silent inconsistency or invented ids would be costly, unless the guardrails above are in place.
Which model for what
| Model | Strengths | Weaknesses | Use it for |
|---|---|---|---|
| Qwen3.6-35B-A3B | Best agent (E4, E1); fast; 28 GB | Weaker on L3 completeness | The default local model; tool loops |
| Qwen3.8-27B | Most accurate on judgement and Q&A (L1, L3, E2) | Slow (21 tokens/s); loops in E1 | Batch screening and Q&A, not interactive agents |
| Gemma 4 26B-A4B | Classification and summaries | Repetition loops as an agent | Classification; never unguarded tool loops |
| Qwen3-Coder-Next | Decent coding with a big budget | 56 GB; burns steps | Not preferred over Qwen3.6 |
| Nemotron 3.5 30B-A3B | Fast decode | Invents ids; weak abstention and consistency | Not recommended for these tasks |
What about cost?
Local cost here uses the study’s amortised €0.03 per GPU-hour. On the small-model-ready tasks, local inference is two to three orders of magnitude cheaper than Opus, and roughly 10 to 80 times cheaper than DeepSeek. Cost rarely decides it, though: accuracy and failure modes do. A cheap model that loops on a quarter of your agent tasks isn’t cheap.
Limits. Opus is missing on E1 and partial on E4, so those verdicts use DeepSeek as the baseline. E4 had one trial per task (about ±2.5 points). E1 is cut down to 40 django and sympy tasks that run on arm64. L2 and E2 severity use synthetic data and are at ceiling, and L3's questions were written by agents and checked separately, not by clinicians. Every local model ran at about 4 bits on llama.cpp; higher precision or another runtime could score differently. Only one model was fine-tuned, on one task.
How the whole study fitted on one box in eight days, and what went wrong along the way, is part 2.
