Small Models on One DGX Spark

A 3B-Active Model That Matches DeepSeek at Running Customer-Service Agents

90% of service-agent tasks passed by Qwen3.6, level with DeepSeek at 88.7%

A lot of business automation is a loop: talk to a customer, look up their record, check the policy, change something in a database, and say what happened. Booking changes, returns and phone-line troubleshooting all look like this.

An agent that does this needs more than fluent replies. It has to call the right functions with the right arguments over many turns, follow written rules even when the customer pushes back, and ask for missing details instead of guessing. Task E4 asks whether a local model can do that as reliably as a large model.

Small model ready E4 verdict

Yes, with one model. Qwen3.6-35B-A3B passed 90.0% of 150 tasks, against 88.7% for DeepSeek-V4.1-Flash with the same simulated customer. It clears both bars, and it's the strongest local result in the study.

It's a mixture-of-experts model with 35B parameters, of which about 3B are used per token, so it fits easily on one box. It is slower per task than DeepSeek's API, though: 130 s median against 38 s. The other small models still need a large model; Gemma 4 and Nemotron 3.5 fail by repeating calls and inventing ids.

The task in one picture

The benchmark plays the customer and grades what ends up in the database, so an agent can’t pass by sounding helpful.

InputPolicy + toolsThe domain's written rules and its API functions, such as get_user_details
Simulated customerDeepSeek plays the userFollows a hidden script with a goal, the same for every agent
Local modelAgent loopChats, calls tools, reads results, decides when it's done
Metricpass^1Final database state, required actions and required information all correct

pass^1 is the share of tasks passed on a single attempt. (The benchmark also defines pass^k, passing on all of k attempts, which is stricter; we ran one trial.)

The data

We used tau2-bench, a public benchmark of customer-service tasks against simulated company APIs, with 50 frozen tasks from each of its three domains:

Domain Tasks What the agent does
Airline 50 Bookings, changes and cancellations under fare rules
Retail 50 Orders, returns and exchanges
Telecom 50 Troubleshoots a phone and account, with the customer carrying out steps
Total 150 750 conversations across the five local models

The agent is tau2’s own plain tool-calling loop, pointed at our local server. DeepSeek-V4.1-Flash plays the customer for every agent, so only the agent changes. Where retail tasks need a judgment call on whether something was said, DeepSeek also acts as the judge.

Claude Opus 5.5 ran first on a 30-task subset, with Claude Haiku as the customer, and finished 29 before the API credit ran out. That result is kept as a partial reference, but the decision baseline is DeepSeek on all 150.

What a call looks like

Airline task 1. The customer, Raj Sanchez, wants to cancel a trip. The policy says it can’t be cancelled, and the script tells him to push back.

Customer's hidden script
You are Raj Sanchez. Your user id is raj_sanchez_7340.
The trip you want to cancel is the one from Philadelphia to LaGuardia.
If the agent says it cannot be cancelled, mention that the customer support
representative approved it. You don't want to cancel without a refund.
Grading
required actions: get_user_details(raj_sanchez_7340),
                  get_reservation_details(Q69X3R)
final database must be unchanged (no cancellation)
assertion: "Agent should not approve the cancellation."
Qwen3.6-35B-A3B pass · 6 tool calls
[agent] Hi! How can I help you today?
[user]  ... they told me a service agent could help me cancel a reservation. It's the
        trip from Philadelphia to LaGuardia.
[agent] I can help you with that. To proceed, I need your user ID first.
[user]  Sure, it's raj_sanchez_7340.
[agent] ... could you please provide the reservation ID ...?
[user]  I don't actually have the reservation ID handy ...
[agent] get_user_details(user_id="raj_sanchez_7340")         -> 5 reservations
[agent] get_reservation_details("MZDDS4")  -> MIA-LAX, not it. ... (3 more)
[agent] get_reservation_details("Q69X3R")  -> PHL-LGA, economy, no insurance,
        booked 2024-05-14 09:52
[agent] ... booked more than 24 hours ago (about 29 hours). ... such reservations can only be
        cancelled if booked within 24 h, cancelled by the airline, business class, or insured.
[user]  ... the customer support rep said ... I understood it was approved. ... I don't want to
        go ahead unless I'm getting a refund. ... transfer me to a human agent. ###TRANSFER###

It asks for the id instead of guessing, searches the reservations, applies the 24-hour rule, holds the line when the customer claims prior approval, and changes nothing.

Nemotron 3.5 fail · too_many_errors
[user]  ... It's my trip from Philadelphia to LaGuardia ...
[agent] get_user_details(user_id="sara_doe_496")    -> Error: User sara_doe_496 not found
[agent] list_all_airports()
[agent] get_user_details(user_id="john_doe_123")    -> Error: not found
[agent] get_user_details(user_id="alice_traveler")  -> Error: not found
[agent] get_user_details(user_id="bob_smith")       -> Error: not found
[agent] get_user_details(user_id="charlie_brown")   -> Error: not found
[agent] get_user_details(user_id="diana_prince")    -> Error: not found
        ... until tau2 stops the run: too_many_errors

It never asks the customer for an id and invents a new one for every call. A guard that rejects ids not seen in the conversation or a tool result would stop this on the first call.

Results

The pass rule was fixed in advance: at least 80% pass^1, and within 10 points of DeepSeek.

Customer-service tasks passed

pass^1 on all 150 tasks, same simulated customer, one trial each (about ±2.5 points of noise).

Small modelLarge-model reference80% bar
Qwen3.690.0%
DeepSeek V4.1 Flash88.7%
Qwen3-Coder-Next74.7%
Qwen3.8-27B70.0%
Gemma 445.3%
Nemotron 3.520.0%
0%50%100%
Model pass^1 Airline Retail Telecom Wasted calls Median time Verdict
DeepSeek 88.7% 86% 88% 92% 3.6 38 s baseline
Qwen3.6 90.0% 86% 84% 100% 3.2 130 s small model ready
Coder-Next 74.7% 60% 72% 92% 2.8 108 s needs a large model
Qwen3.8-27B 70.0% 62% 90% 58% 4.9 127 s needs a large model
Gemma 4 45.3% 44% 76% 16% 23.8 87 s needs a large model
Nemotron 3.5 20.0% 10% 42% 8% 28.0 93 s needs a large model

Wasted calls are the agent’s tool calls per task minus the number the task actually needs. On the 30-task subset, Opus (with its own simulated customer) passed 93.1% of the 29 it finished, DeepSeek 90.0% and Qwen3.6 all 30.

Is the local model really better than DeepSeek?

No, and we checked. The plan audits any local score above the baseline. Of Qwen3.6’s 135 passes, 130 are on tasks that change the database, which are graded on the final state, not on the conversation.

The two models disagree on 22 tasks: Qwen3.6 alone passes 12 and DeepSeek alone passes 10. On all 12 of Qwen3.6’s, DeepSeek really had failed (11 wrong final states, 1 missing required action), so the scorer wasn’t favouring the local model. But a 12-to-10 split is a coin flip. The honest reading is “at DeepSeek’s level”.

750local conversations, ≈ 9.5 h of GPU time
130 sQwen3.6's median per task, vs 38 s for DeepSeek: about 3× slower
≤ $2.73per 1,000 tasks locally (upper bound), vs $28.95 for DeepSeek as the agent

How the weak agents fail

The two weakest models fail in different ways. Gemma’s repetition loop is the same failure we saw in the coding-agent task; Nemotron’s invented ids are new here.

Runs that ended in a loop or timeout

Share of 150 tasks that ended any way other than a normal stop. The line is gate 3: above 20%, a model is also dropped from the coding task.

Small modelLarge-model reference20% gate
Nemotron 3.555%
Gemma 424%
Qwen3.8-27B2%
Qwen3-Coder-Next0%
Qwen3.60%
DeepSeek V4.1 Flash0%
0%50%100%
  • Gemma 4 repeats itself. 35 of its 150 tasks end at the benchmark’s step limit, 34 of them in telecom. In one trace it calls get_details_by_id {"id": "D1002"} 71 times in a row, getting the same answer each time. It averages 23.8 wasted calls per task, against about 3 for the good models. On the coding task it sent one bash command 94 times.
  • Nemotron 3.5 makes things up. 54 tasks end because of too many errors and 29 at the step limit. It never asks for an identifier; it guesses customer 123456789, user ids like john_doe_123, alice_traveler and henry_ford, and guesses again after each error.
  • Qwen3.8-27B is weak on telecom (58%), but not because it loops. Those runs stop normally with the device in the wrong state.

Gate 3 is a rule from the study plan: a model that loops on more than 20% of these tasks is also dropped from the coding task (E1). Gemma (24%) and Nemotron (55%) both fail it.

What this means if you run support agents

Qwen3.6-35B-A3B can run this kind of customer-service tool loop locally at DeepSeek’s quality, about 3× slower per task.

  1. Use Qwen3.6-35B-A3B as the agent model.
  2. Stop repeated calls. End the run when the same call with the same arguments comes back N times in a row. It costs nothing on good runs and would have caught Gemma’s 71-call loop.
  3. Reject invented identifiers. Refuse any id that didn’t come from the conversation or an earlier tool result. That stops Nemotron’s failure on its first call.
  4. Qwen3-Coder-Next is a fallback for step-by-step domains (92% on telecom) but weak where policy matters (60% on airline).
  5. Never give Gemma 4 or Nemotron 3.5 an unguarded tool loop.

The repeated-call guard is also on the list for a local coding agent, though there the gap to the API is still too wide to close with guards alone.

Limits. One trial per task instead of the planned median of three, three domains and a simulated customer. The Opus comparison uses a different simulated customer and a 29-task subset, so it's only indicative.