Small Models on One DGX Spark

Fixing Real Bugs in Real Repos: Where Small Models Still Trail by 22.5 Points

22.5 points behind DeepSeek on real bugs: resolved-task bars for DeepSeek and four local models

A coding agent is the hardest everyday job you can give a language model. It has to read a bug report, find its way around a repository it has never seen, edit the right file, run the tests and then stop. Each step depends on the last, so one bad move early on can sink the whole attempt.

It’s also the job teams most want to bring in-house, because code is the data they least like sending to an API. Task E1 asks a narrow question: can a model running on one desk-side box close well-described issues in real open-source projects often enough to be useful?

Needs a large model E1 verdict

Not yet. DeepSeek-V4.1-Flash fixed 38 of 40 real issues (95%). The best local model, Qwen3.6-35B-A3B, fixed 29 (72.5%). That clears the 50% absolute bar but sits 22.5 points behind, outside the allowed 15.

The local models handle small fixes well and fall away as a task needs more exploring. This is the largest small-vs-large gap in the study. Claude Opus 5.5 has no result here: its API credit ran out before E1, so DeepSeek is the baseline.

The task in one picture

The model drives a standard open-source coding agent inside a container. It never sees the tests that grade it.

InputRepo + issueThe project checked out at the commit before the fix, plus the issue text
Local modelAgent loopOpenCode's bash, read, grep and edit tools; up to 100 steps and 30 minutes
CodeTake the diffgit diff against the starting commit, applied in a fresh container
MetricHidden testsSWE-bench's own grader: the failing tests must now pass and the rest must still pass

A step is one turn of the loop: the model either calls a tool or answers. The step cap is how many turns it gets before the agent is cut off.

The data

The tasks come from SWE-bench Verified, a set of 500 real GitHub issues that humans checked could be solved from the issue text. Each comes with the maintainers’ fix and the tests that prove it.

Item Detail
Tasks 40: 31 from Django, 9 from SymPy
Difficulty (SWE-bench’s own label) 16 under 15 min, 20 from 15 min to 1 h, 4 from 1 to 4 h
Hidden tests 85 that must start passing, 4,222 that must keep passing
Validation Every task fails with no patch and passes with the maintainers’ patch on our images
Grading SWE-bench’s own code (swebench 5.0.2), with no pass/fail logic of ours

The sample keeps Verified’s mix of difficulty and leaves out the few tasks labelled over 4 hours. It is limited to Django and SymPy for a practical reason. SWE-bench only publishes x86 container images and the Spark is an Arm machine, so we built 19 Arm environments ourselves. Both projects are pure Python, and between them they cover 306 of the 500 Verified tasks.

What a call looks like

The prompt is short and the same for every task. Web access and sub-agents are switched off, so the model can’t look up the real fix.

Prompt
You are working in the git repository checked out at /testbed (sympy/sympy).
Resolve the following issue by editing the source code in /testbed.
<issue> ... </issue>
Instructions:
- Implement a fix in the repository's source code (not in the tests).
- You may read code and run Python or the existing test suite (e.g.
  cd /testbed && python bin/test sympy/<module>/tests/test_<x>.py).
- Do not modify or delete existing tests. Do not commit.
- There is no internet access. When the fix is done, stop and reply with a
  one-line summary.
Issue · sympy-20801 · 15 min to 1 h
>>> from sympy import *
>>> S(0.0) == S.false
True
>>> S.false == S(0.0)
False
Based on the results of comparison to `S(0)`:
>>> S(0) == S.false
False
>>> S.false == S(0)
False
I assume we would want `S(0.0) == S.false` to return True as well?
Gold

There's a trap. The issue suggests True, but the maintainers made both comparisons return False, because a floating-point zero is not a Boolean. The hidden tests check the maintainers' behaviour.

Qwen3.6-35B-A3B · 18 turns: 10 bash, 4 read, 2 grep, 1 edit
--- a/sympy/core/numbers.py
+++ b/sympy/core/numbers.py
@@ -1386,10 +1386,10 @@ def __eq__(self, other):
             other = _sympify(other)
         except SympifyError:
             return NotImplemented
-        if not self:
-            return not other
         if isinstance(other, Boolean):
             return False
+        if not self:
+            return not other
         if other.is_NumberSymbol:

It moves the zero check below the Boolean check, so S(0.0) == S.false now returns False. The maintainers moved the same two lines further down; the tests can't tell the two apart, and the grader marks it resolved.

Five of the six models fixed this one. The interesting differences are in how they got there:

Model Resolved Turns Wall time
DeepSeek-V4.1-Flash yes 40 204 s
Qwen3.6-35B-A3B yes 18 121 s
Qwen3.8-27B yes 59 982 s
Nemotron 3.5 30B-A3B yes 40 234 s
Qwen3-Coder-Next yes 100 (cap) 1,094 s
Gemma 4 26B-A4B no 21 532 s

Qwen3-Coder-Next had a correct fix and then kept re-running tests until it hit the step cap. The diff counts, but it took nine times as long as Qwen3.6.

Results

The headline metric is pass@1: the share of tasks where the agent’s single attempt resolves the issue under SWE-bench’s grader. The pass rule was fixed in advance: at least 50%, and within 15 points of DeepSeek (so at least 80% here).

Real issues resolved, out of 40

pass@1 at the 100-step cap (the decision pass) and at the plan's original 20-step cap. The line is the 50% absolute bar; the relative bar is 80%.

Small modelLarge-model reference50% bar
Step cap 100
DeepSeek V4.1 Flash95.0%
Qwen3.672.5%
Qwen3.8-27B62.5%
Qwen3-Coder-Next60.0%
Nemotron 3.552.5%
Gemma 425.0%
Step cap 20
DeepSeek V4.1 Flash82.5%
Qwen3.8-27B55.0%
Qwen3.650.0%
Gemma 437.5%
Nemotron 3.532.5%
Qwen3-Coder-Next17.5%
0%50%100%
Model pass@1 Quick Medium Hard Hit cap Median time Verdict
DeepSeek 95.0% 16/16 18/20 4/4 0% 56 s baseline
Qwen3.6 72.5% 15/16 12/20 2/4 15% 194 s needs a large model
Qwen3.8-27B 62.5% 13/16 11/20 1/4 15% 305 s needs a large model
Coder-Next 60.0% 14/16 10/20 0/4 60% 1,102 s needs a large model
Nemotron 3.5 52.5% 12/16 9/20 0/4 28% 320 s needs a large model
Gemma 4 25.0% 8/16 2/20 0/4 7% 223 s needs a large model

All figures are at the 100-step cap. Quick, medium and hard are SWE-bench’s under-15-minute, 15-minute-to-1-hour and 1-to-4-hour labels. Read across the difficulty columns and the pattern is plain: on the quickest fixes Qwen3.6 is one task behind DeepSeek, on medium ones six behind, and on the hardest it solves two of four.

Why there are two step caps

The plan set a 20-step cap. In that first pass, 57–97% of local runs were cut off mid-fix, so the cap was measuring patience rather than skill. We raised it to 100 and report both passes. The verdict is the same under either cap, and at 20 steps the gaps are wider.

480agent runs: 400 local, 80 on DeepSeek, across both caps
≈ 11 hGPU time: ≈ 3 h at cap 20, ≈ 8 h at cap 100
3.5×Qwen3.6's median time per task against DeepSeek's (194 s vs 56 s)
≤ $3.59per 1,000 tasks locally (upper bound), vs $124 on DeepSeek

What the traces showed

We read the agent logs for every surprising result. Five failure patterns stand out. The loops and the step-budget problems come back in the customer-service agent task, where a sixth one appears: models inventing ids instead of asking for them.

  • Repetition loops. Gemma 4 and Qwen3.8 each had 8 tasks where one identical tool call repeated 10 or more times. One Gemma run sent the same bash command 94 times; one Qwen3.8 run hit 97. DeepSeek and the other local models had none.
  • More room isn’t always better. Every model fixed more with 100 steps, except Gemma, which dropped from 15 to 10: given room, it fell into loops. Coder-Next went from 7 to 24, and all 24 needed more than 20 turns.
  • Not knowing when to stop. Coder-Next’s median run used all 100 steps. In 13 of its resolved tasks, the correct diff was already written when the cap ended the run; it was just re-running tests.
  • No patch at all. 20% of Qwen3.8’s runs ended with an empty diff.
  • Breaking the rules quietly. On django-15315, Qwen3.6 also edited an existing test to match its fix, against the prompt. SWE-bench restores test files before grading, so it still counts as resolved. A human reviewer would reject that part of the diff.

DeepSeek’s two failures are genuinely wrong fixes, and no local model solves those tasks either. Ten tasks were solved by all five local models, nine by none of them.

We also caught a bug of our own. The first grading of the 100-step pass reused cached grades from the 20-step pass, because the cache was keyed by model and task rather than by the patch. It showed no model losing any task and counted one empty patch as a fix. The cache is now keyed by a hash of the patch, and all 240 predictions were regraded.

Gate 3: models that loop are out

The study plan has a rule, called gate 3, that drops any model that loops on more than 20% of the customer-service agent tasks from coding. Gemma 4 (24%) and Nemotron 3.5 (55%) fail it. E1 ran before that task finished, so both have results above, but they were never candidates. Qwen3.8 has 8 looping tasks here, but loops or times out on only 2% of the agent tasks, so it stays in.

What this means if you want a local coding agent

  1. Keep repository-level coding on a large model for now, through an API or a large open model you host. The gap is widest exactly where agents earn their keep: issues that need exploring.
  2. For small, well-scoped fixes, use Qwen3.6-35B-A3B. It solved 15 of 16 of the quickest tasks.
  3. Give it a generous step budget, at least 100. At 20 you measure the cap, not the model.
  4. Add a repeated-call guard that stops the run when the same call with the same arguments repeats. It costs nothing on good runs.
  5. Review every diff, including the test files. The grader won’t catch a model that rewrote a test to agree with it.
  6. Don’t use Gemma 4 or Nemotron 3.5 for agent loops of any kind.

Limits. 40 tasks from two pure-Python projects, one run per task, so each figure carries roughly ±7 points of sampling noise. No Opus result. Local times include four agents sharing the server at once, so the local cost is an upper bound.