Before Any Benchmark: Can Five Open Models Run Well on One DGX Spark?
An evaluation is only as good as the serving setup under it. If the server quietly cuts the front off a long prompt, or a tool call comes back malformed, every later score still looks plausible, and every later score is wrong.
So before spending GPU days on eight real tasks, we asked a plainer question: can each candidate model be served on one NVIDIA DGX Spark, at a usable speed, without silent failures? This is task T01, the study’s first gate. A model that failed here would not be worth testing further.
All five models entered the study. Every one loaded, returned a code hidden at the start of a 32k-token prompt, completed a 5-call tool chain and leaked no reasoning text. The four mixture-of-experts models decode at 52–66 tok/s and pass outright.
Qwen3.8-27B, the only dense model, decodes at 21 tok/s, under the 25 tok/s bar. It went forward as a conditional pass: fine for batch work, slow for interactive agents.
The gate in one picture
Each model is brought up on the same server with the rest of the box idle, then put through five short probes. Nothing here tests task skill; it tests whether the plumbing can be trusted.
The models and the probes
Five open models, all under 100B parameters, each in a 4-bit quantisation (weights stored in about 4 bits instead of 16, so they fit in memory). Four are mixture-of-experts (MoE) models: they hold many parameters but use only 3–4B of them for each token, which makes them fast. Qwen3.8-27B is dense: all 27B parameters work on every token.
| Model | Type | Quantisation |
|---|---|---|
| Qwen3.6-35B-A3B | MoE, 3B active | UD-Q4_K_M |
| Nemotron 3.5 30B-A3B | MoE, 3B active | UD-Q4_K_M |
| Gemma 4 26B-A4B | MoE, 4B active | MXFP4 |
| Qwen3-Coder-Next | MoE, 80B total, 3B active | UD-Q4_K_XL |
| Qwen3.8-27B | Dense | Q4_K_M, with an MTP draft head |
The hardware is one DGX Spark: a GB10 chip with 121 GB of unified memory (the CPU and GPU share one pool), running llama.cpp build 10851. The whole gate took about 2 hours on 31 October 2026, downloads included.
The probes, in plain terms:
- Decode speed is how fast the model writes, in tokens per second. Prefill speed is how fast it reads the prompt. Both are measured on a ~2k- and a ~32k-token prompt made of unique random words, so the server’s prompt cache can’t help.
- 4-stream throughput is total decode speed with four requests at once.
- Peak memory is the drop in available memory from idle: weights, cache and working buffers together.
- The 32k needle hides a 6-digit code at the very start of a long prompt and asks for it at the end. If the server truncated the prompt, the code is gone.
- Stray reasoning checks that no hidden “thinking” text leaks into answers when thinking is switched off.
What a call looks like
The two correctness probes are deterministic. The tools are fakes that only answer when called with the right arguments, so there is exactly one way to get the answer.
You are an operations assistant. Use the tools; call one tool at a time.
Find Marta Kowalski's only order, work out its total cost in EUR (qty × unit price), then save a note on her account saying 'Order total: <amount> EUR'. Finally tell me the total.
find_user(name) → {"user_id": "U-4471"} (only for Marta)
get_orders(user_id) → {"orders": ["O-9002"]}
get_order(order_id) → {"sku": "SKU-77", "qty": 3}
get_price(sku) → {"eur": 12.5}
write_note(user_id, note) → {"ok": true}All five tools called, a note containing 37.5, and 37.5 in the final answer (3 × 12.5 EUR).
calls: find_user → get_orders → get_order → get_price → write_note (5 calls) note: "Order total: 37.5 EUR" final: "The total cost of Marta Kowalski's order is 37.5 EUR."
The same 5 calls in the same order, the correct note and answer, and no reasoning text leaked.
The needle prompt reads: “Remember this: the vault code is <6 digits>”, then about 10,200 random filler words (roughly 32k tokens), then “What is the vault code given at the very start? Reply with the 6 digits only.” All five models returned the code. We also checked the token count the server reported back (31.6k–37.8k, depending on each model’s tokenizer), so we know nothing was cut from the front.
Results: four fast models and one slow one
The speed bar was 25 tok/s single-stream decode, set before any model ran.
Single-stream decode speed
Tokens per second, greedy, 256 generated tokens. Scale 0–100 tok/s.
| Model | Decode 2k | Decode 32k | Prefill 2k | Prefill 32k | 4-stream | Peak GB | Gate 1 |
|---|---|---|---|---|---|---|---|
| Qwen3.6-35B-A3B | 66.1 | 54.8 | 2,463 | 2,344 | 87.3 | 27.6 | pass |
| Nemotron 3.5 30B-A3B | 64.9 | 60.7 | 1,715 | 2,342 | 79.1 | 28.2 | pass |
| Qwen3-Coder-Next | 52.2 | 43.2 | 1,370 | 1,643 | 64.6 | 55.7 | pass |
| Gemma 4 26B-A4B | 51.5 | 41.6 | 2,567 | 2,911 | 88.6 | 27.6 | pass |
| Qwen3.8-27B (dense) | 21.0 | 18.5 | 752 | 724 | 30.7 | 42.4 | conditional |
All speeds are tokens per second. Every model also passed the needle, the tool chain (exactly 5 calls each) and the stray-reasoning check.
The MoE models lose 6–19% of their decode speed at 32k tokens and still sit well above the bar. Gemma 4 measured 51.5 tok/s, below a public figure of 69.9; that figure was for a different quantisation (Q4_K_M) on an older build.
Peak memory, as a share of the Spark's 121 GB
Weights, cache and buffers together, one model loaded. Bars show the share of 121 GB; the numbers are GB.
Three of the MoE models peak at about 28 GB, so one of them leaves most of the box free for other services. Qwen3-Coder-Next is the largest footprint at 55.7 GB, because it stores 80B parameters in total, and it still passes comfortably at 52 tok/s.
Why the dense model is slow, and why it stayed in
Decoding a dense 27B model means reading all of its weights for every token, so its speed is limited by memory bandwidth rather than compute. An MoE model with 3–4B active parameters reads far less per token.
Qwen3.8-27B was served with an MTP draft head, a small built-in predictor that guesses several tokens ahead so the main model can check them in one pass. How much that helps depends on the text. On the free-prose speed prompt only 39% of the guesses were accepted (2.2 tokens per accepted run on average). On tool calls and short structured turns, acceptance was 87–100% (runs of 3.6–4.0 tokens). The draft head also keeps its own cache per slot, which is why this model peaks at 42 GB.
The study plan said to judge speed on real task runs, not on this probe alone. On the L2 report-structuring task (JSON output, 4 streams at once), Qwen3.8 decoded at 20.2 tok/s per stream against 22.7–28.9 for the MoE models. That’s slower, but workable for batch jobs, so it stayed in on the same llama.cpp setup, flagged slow. The fallback plan (a different serving stack, SGLang) was not needed.
That turned out to matter. Qwen3.8-27B went on to be the most accurate local model on L1 screening, L3 guideline Q&A and E2 log triage. It was also the slowest, and took about twice the wall time per coding task on E1.
What the probes caught, and what they didn’t
Nothing failed silently. No model truncated the 32k prompt, every model finished the tool chain in exactly five calls, and none leaked reasoning text. The only gate-1 miss was speed, and it was a known trade-off of dense models, not a fault.
Two measurement details made the numbers trustworthy:
- Unique-token prompts. A repeated prompt measures the server’s prefix cache, not the model. Random unique words force a real prefill every time.
- Checking the echoed token count. Silent truncation is the most dangerous serving failure, because every later score still looks plausible. Comparing the token count the server reports with the prompt we sent catches it.
What this means if you serve models on a Spark
- Pick an MoE model for anything interactive. With 3–4B active parameters they decode at 52–66 tok/s and use about 28 GB.
- Treat a dense 27B as a batch model. At about 20 tok/s it’s fine for overnight extraction or screening, and slow for chat or agent loops.
- Probe speculative decoding on your real workload. The same draft head gave 39% acceptance on prose and 87–100% on tool calls.
- Run a needle test and check
prompt_tokensbefore trusting any long-context result. - Budget memory per model, not per box. Even the 55.7 GB Coder-Next leaves room on 121 GB for other services.
Limits. These are single-machine probes with the rest of the box idle, at one quantisation per model and one llama.cpp build. Load times were recorded but are not comparable across models (Gemma 4 was already resident from a warm-up). Qwen3-Coder-Next was tested after the L2 batch because its download finished mid-phase. Speed on real tasks varies with prompt length and output shape; see each task post for measured throughput.
