A 3B-Active Model That Matches DeepSeek at Running Customer-Service Agents
A lot of business automation is a loop: talk to a customer, look up their record, check the policy, change something in a database, and say what happened. Booking changes, returns and phone-line troubleshooting all look like this.
An agent that does this needs more than fluent replies. It has to call the right functions with the right arguments over many turns, follow written rules even when the customer pushes back, and ask for missing details instead of guessing. Task E4 asks whether a local model can do that as reliably as a large model.
Yes, with one model. Qwen3.6-35B-A3B passed 90.0% of 150 tasks, against 88.7% for DeepSeek-V4.1-Flash with the same simulated customer. It clears both bars, and it's the strongest local result in the study.
It's a mixture-of-experts model with 35B parameters, of which about 3B are used per token, so it fits easily on one box. It is slower per task than DeepSeek's API, though: 130 s median against 38 s. The other small models still need a large model; Gemma 4 and Nemotron 3.5 fail by repeating calls and inventing ids.
The task in one picture
The benchmark plays the customer and grades what ends up in the database, so an agent can’t pass by sounding helpful.
pass^1 is the share of tasks passed on a single attempt. (The benchmark also defines pass^k, passing on all of k attempts, which is stricter; we ran one trial.)
The data
We used tau2-bench, a public benchmark of customer-service tasks against simulated company APIs, with 50 frozen tasks from each of its three domains:
| Domain | Tasks | What the agent does |
|---|---|---|
| Airline | 50 | Bookings, changes and cancellations under fare rules |
| Retail | 50 | Orders, returns and exchanges |
| Telecom | 50 | Troubleshoots a phone and account, with the customer carrying out steps |
| Total | 150 | 750 conversations across the five local models |
The agent is tau2’s own plain tool-calling loop, pointed at our local server. DeepSeek-V4.1-Flash plays the customer for every agent, so only the agent changes. Where retail tasks need a judgment call on whether something was said, DeepSeek also acts as the judge.
Claude Opus 5.5 ran first on a 30-task subset, with Claude Haiku as the customer, and finished 29 before the API credit ran out. That result is kept as a partial reference, but the decision baseline is DeepSeek on all 150.
What a call looks like
Airline task 1. The customer, Raj Sanchez, wants to cancel a trip. The policy says it can’t be cancelled, and the script tells him to push back.
You are Raj Sanchez. Your user id is raj_sanchez_7340. The trip you want to cancel is the one from Philadelphia to LaGuardia. If the agent says it cannot be cancelled, mention that the customer support representative approved it. You don't want to cancel without a refund.
required actions: get_user_details(raj_sanchez_7340),
get_reservation_details(Q69X3R)
final database must be unchanged (no cancellation)
assertion: "Agent should not approve the cancellation."[agent] Hi! How can I help you today?
[user] ... they told me a service agent could help me cancel a reservation. It's the
trip from Philadelphia to LaGuardia.
[agent] I can help you with that. To proceed, I need your user ID first.
[user] Sure, it's raj_sanchez_7340.
[agent] ... could you please provide the reservation ID ...?
[user] I don't actually have the reservation ID handy ...
[agent] get_user_details(user_id="raj_sanchez_7340") -> 5 reservations
[agent] get_reservation_details("MZDDS4") -> MIA-LAX, not it. ... (3 more)
[agent] get_reservation_details("Q69X3R") -> PHL-LGA, economy, no insurance,
booked 2024-05-14 09:52
[agent] ... booked more than 24 hours ago (about 29 hours). ... such reservations can only be
cancelled if booked within 24 h, cancelled by the airline, business class, or insured.
[user] ... the customer support rep said ... I understood it was approved. ... I don't want to
go ahead unless I'm getting a refund. ... transfer me to a human agent. ###TRANSFER###
It asks for the id instead of guessing, searches the reservations, applies the 24-hour rule, holds the line when the customer claims prior approval, and changes nothing.
[user] ... It's my trip from Philadelphia to LaGuardia ...
[agent] get_user_details(user_id="sara_doe_496") -> Error: User sara_doe_496 not found
[agent] list_all_airports()
[agent] get_user_details(user_id="john_doe_123") -> Error: not found
[agent] get_user_details(user_id="alice_traveler") -> Error: not found
[agent] get_user_details(user_id="bob_smith") -> Error: not found
[agent] get_user_details(user_id="charlie_brown") -> Error: not found
[agent] get_user_details(user_id="diana_prince") -> Error: not found
... until tau2 stops the run: too_many_errors
It never asks the customer for an id and invents a new one for every call. A guard that rejects ids not seen in the conversation or a tool result would stop this on the first call.
Results
The pass rule was fixed in advance: at least 80% pass^1, and within 10 points of DeepSeek.
Customer-service tasks passed
pass^1 on all 150 tasks, same simulated customer, one trial each (about ±2.5 points of noise).
| Model | pass^1 | Airline | Retail | Telecom | Wasted calls | Median time | Verdict |
|---|---|---|---|---|---|---|---|
| DeepSeek | 88.7% | 86% | 88% | 92% | 3.6 | 38 s | baseline |
| Qwen3.6 | 90.0% | 86% | 84% | 100% | 3.2 | 130 s | small model ready |
| Coder-Next | 74.7% | 60% | 72% | 92% | 2.8 | 108 s | needs a large model |
| Qwen3.8-27B | 70.0% | 62% | 90% | 58% | 4.9 | 127 s | needs a large model |
| Gemma 4 | 45.3% | 44% | 76% | 16% | 23.8 | 87 s | needs a large model |
| Nemotron 3.5 | 20.0% | 10% | 42% | 8% | 28.0 | 93 s | needs a large model |
Wasted calls are the agent’s tool calls per task minus the number the task actually needs. On the 30-task subset, Opus (with its own simulated customer) passed 93.1% of the 29 it finished, DeepSeek 90.0% and Qwen3.6 all 30.
Is the local model really better than DeepSeek?
No, and we checked. The plan audits any local score above the baseline. Of Qwen3.6’s 135 passes, 130 are on tasks that change the database, which are graded on the final state, not on the conversation.
The two models disagree on 22 tasks: Qwen3.6 alone passes 12 and DeepSeek alone passes 10. On all 12 of Qwen3.6’s, DeepSeek really had failed (11 wrong final states, 1 missing required action), so the scorer wasn’t favouring the local model. But a 12-to-10 split is a coin flip. The honest reading is “at DeepSeek’s level”.
How the weak agents fail
The two weakest models fail in different ways. Gemma’s repetition loop is the same failure we saw in the coding-agent task; Nemotron’s invented ids are new here.
Runs that ended in a loop or timeout
Share of 150 tasks that ended any way other than a normal stop. The line is gate 3: above 20%, a model is also dropped from the coding task.
- Gemma 4 repeats itself. 35 of its 150 tasks end at the benchmark’s
step limit, 34 of them in telecom. In one trace it calls
get_details_by_id {"id": "D1002"}71 times in a row, getting the same answer each time. It averages 23.8 wasted calls per task, against about 3 for the good models. On the coding task it sent one bash command 94 times. - Nemotron 3.5 makes things up. 54 tasks end because of too many errors
and 29 at the step limit. It never asks for an identifier; it guesses
customer
123456789, user ids likejohn_doe_123,alice_travelerandhenry_ford, and guesses again after each error. - Qwen3.8-27B is weak on telecom (58%), but not because it loops. Those runs stop normally with the device in the wrong state.
Gate 3 is a rule from the study plan: a model that loops on more than 20% of these tasks is also dropped from the coding task (E1). Gemma (24%) and Nemotron (55%) both fail it.
What this means if you run support agents
Qwen3.6-35B-A3B can run this kind of customer-service tool loop locally at DeepSeek’s quality, about 3× slower per task.
- Use Qwen3.6-35B-A3B as the agent model.
- Stop repeated calls. End the run when the same call with the same arguments comes back N times in a row. It costs nothing on good runs and would have caught Gemma’s 71-call loop.
- Reject invented identifiers. Refuse any id that didn’t come from the conversation or an earlier tool result. That stops Nemotron’s failure on its first call.
- Qwen3-Coder-Next is a fallback for step-by-step domains (92% on telecom) but weak where policy matters (60% on airline).
- Never give Gemma 4 or Nemotron 3.5 an unguarded tool loop.
The repeated-call guard is also on the list for a local coding agent, though there the gap to the API is still too wide to close with guards alone.
Limits. One trial per task instead of the planned median of three, three domains and a simulated customer. The Opus comparison uses a different simulated customer and a 29-task subset, so it's only indicative.
