Small Models on One DGX Spark

Before Any Benchmark: Can Five Open Models Run Well on One DGX Spark?

Decode-speed bars: four MoE models at 52 to 66 tokens per second, the dense 27B model at 21

An evaluation is only as good as the serving setup under it. If the server quietly cuts the front off a long prompt, or a tool call comes back malformed, every later score still looks plausible, and every later score is wrong.

So before spending GPU days on eight real tasks, we asked a plainer question: can each candidate model be served on one NVIDIA DGX Spark, at a usable speed, without silent failures? This is task T01, the study’s first gate. A model that failed here would not be worth testing further.

Pass T01 verdict (gate 1)

All five models entered the study. Every one loaded, returned a code hidden at the start of a 32k-token prompt, completed a 5-call tool chain and leaked no reasoning text. The four mixture-of-experts models decode at 52–66 tok/s and pass outright.

Qwen3.8-27B, the only dense model, decodes at 21 tok/s, under the 25 tok/s bar. It went forward as a conditional pass: fine for batch work, slow for interactive agents.

The gate in one picture

Each model is brought up on the same server with the rest of the box idle, then put through five short probes. Nothing here tests task skill; it tests whether the plumbing can be trusted.

SetupOne model at a timellama.cpp in router mode, 4 parallel slots, reasoning off
Local modelSpeed and memoryDecode and prefill at 2k and 32k tokens, 4 concurrent streams, peak memory
Local modelCorrectness probes32k needle, 5-call tool chain, stray-reasoning check
Gate 1Pass or not≥ 25 tok/s decode, no truncation, working tools, no leaked reasoning

The models and the probes

Five open models, all under 100B parameters, each in a 4-bit quantisation (weights stored in about 4 bits instead of 16, so they fit in memory). Four are mixture-of-experts (MoE) models: they hold many parameters but use only 3–4B of them for each token, which makes them fast. Qwen3.8-27B is dense: all 27B parameters work on every token.

Model Type Quantisation
Qwen3.6-35B-A3B MoE, 3B active UD-Q4_K_M
Nemotron 3.5 30B-A3B MoE, 3B active UD-Q4_K_M
Gemma 4 26B-A4B MoE, 4B active MXFP4
Qwen3-Coder-Next MoE, 80B total, 3B active UD-Q4_K_XL
Qwen3.8-27B Dense Q4_K_M, with an MTP draft head

The hardware is one DGX Spark: a GB10 chip with 121 GB of unified memory (the CPU and GPU share one pool), running llama.cpp build 10851. The whole gate took about 2 hours on 31 October 2026, downloads included.

The probes, in plain terms:

  • Decode speed is how fast the model writes, in tokens per second. Prefill speed is how fast it reads the prompt. Both are measured on a ~2k- and a ~32k-token prompt made of unique random words, so the server’s prompt cache can’t help.
  • 4-stream throughput is total decode speed with four requests at once.
  • Peak memory is the drop in available memory from idle: weights, cache and working buffers together.
  • The 32k needle hides a 6-digit code at the very start of a long prompt and asks for it at the end. If the server truncated the prompt, the code is gone.
  • Stray reasoning checks that no hidden “thinking” text leaks into answers when thinking is switched off.

What a call looks like

The two correctness probes are deterministic. The tools are fakes that only answer when called with the right arguments, so there is exactly one way to get the answer.

System prompt
You are an operations assistant. Use the tools; call one tool at a time.
User message
Find Marta Kowalski's only order, work out its total cost in EUR
(qty × unit price), then save a note on her account saying
'Order total: <amount> EUR'. Finally tell me the total.
Tools
find_user(name)           → {"user_id": "U-4471"}   (only for Marta)
get_orders(user_id)       → {"orders": ["O-9002"]}
get_order(order_id)       → {"sku": "SKU-77", "qty": 3}
get_price(sku)            → {"eur": 12.5}
write_note(user_id, note) → {"ok": true}
Pass rule

All five tools called, a note containing 37.5, and 37.5 in the final answer (3 × 12.5 EUR).

Qwen3.6 output pass
calls: find_user → get_orders → get_order → get_price → write_note  (5 calls)
note:  "Order total: 37.5 EUR"
final: "The total cost of Marta Kowalski's order is 37.5 EUR."
All models pass

The same 5 calls in the same order, the correct note and answer, and no reasoning text leaked.

The needle prompt reads: “Remember this: the vault code is <6 digits>”, then about 10,200 random filler words (roughly 32k tokens), then “What is the vault code given at the very start? Reply with the 6 digits only.” All five models returned the code. We also checked the token count the server reported back (31.6k–37.8k, depending on each model’s tokenizer), so we know nothing was cut from the front.

Results: four fast models and one slow one

The speed bar was 25 tok/s single-stream decode, set before any model ran.

Single-stream decode speed

Tokens per second, greedy, 256 generated tokens. Scale 0–100 tok/s.

Small model25 tok/s bar
~2k-token prompt
Qwen3.666.1
Nemotron 3.564.9
Qwen3-Coder-Next52.2
Gemma 451.5
Qwen3.8-27B21.0
~32k-token prompt
Qwen3.654.8
Nemotron 3.560.7
Qwen3-Coder-Next43.2
Gemma 441.6
Qwen3.8-27B18.5
0255075100
Model Decode 2k Decode 32k Prefill 2k Prefill 32k 4-stream Peak GB Gate 1
Qwen3.6-35B-A3B 66.1 54.8 2,463 2,344 87.3 27.6 pass
Nemotron 3.5 30B-A3B 64.9 60.7 1,715 2,342 79.1 28.2 pass
Qwen3-Coder-Next 52.2 43.2 1,370 1,643 64.6 55.7 pass
Gemma 4 26B-A4B 51.5 41.6 2,567 2,911 88.6 27.6 pass
Qwen3.8-27B (dense) 21.0 18.5 752 724 30.7 42.4 conditional

All speeds are tokens per second. Every model also passed the needle, the tool chain (exactly 5 calls each) and the stray-reasoning check.

The MoE models lose 6–19% of their decode speed at 32k tokens and still sit well above the bar. Gemma 4 measured 51.5 tok/s, below a public figure of 69.9; that figure was for a different quantisation (Q4_K_M) on an older build.

Peak memory, as a share of the Spark's 121 GB

Weights, cache and buffers together, one model loaded. Bars show the share of 121 GB; the numbers are GB.

Small model
Qwen3.627.6
Gemma 427.6
Nemotron 3.528.2
Qwen3.8-27B42.4
Qwen3-Coder-Next55.7
0%25%50%75%100%

Three of the MoE models peak at about 28 GB, so one of them leaves most of the box free for other services. Qwen3-Coder-Next is the largest footprint at 55.7 GB, because it stores 80B parameters in total, and it still passes comfortably at 52 tok/s.

52–66tok/s decode for the four MoE models at 2k tokens
21.0tok/s for the dense Qwen3.8-27B, under the 25 bar
5 / 5models passed the 32k needle and the 5-call tool chain
≈ 2 hfor the whole gate, downloads included

Why the dense model is slow, and why it stayed in

Decoding a dense 27B model means reading all of its weights for every token, so its speed is limited by memory bandwidth rather than compute. An MoE model with 3–4B active parameters reads far less per token.

Qwen3.8-27B was served with an MTP draft head, a small built-in predictor that guesses several tokens ahead so the main model can check them in one pass. How much that helps depends on the text. On the free-prose speed prompt only 39% of the guesses were accepted (2.2 tokens per accepted run on average). On tool calls and short structured turns, acceptance was 87–100% (runs of 3.6–4.0 tokens). The draft head also keeps its own cache per slot, which is why this model peaks at 42 GB.

The study plan said to judge speed on real task runs, not on this probe alone. On the L2 report-structuring task (JSON output, 4 streams at once), Qwen3.8 decoded at 20.2 tok/s per stream against 22.7–28.9 for the MoE models. That’s slower, but workable for batch jobs, so it stayed in on the same llama.cpp setup, flagged slow. The fallback plan (a different serving stack, SGLang) was not needed.

That turned out to matter. Qwen3.8-27B went on to be the most accurate local model on L1 screening, L3 guideline Q&A and E2 log triage. It was also the slowest, and took about twice the wall time per coding task on E1.

What the probes caught, and what they didn’t

Nothing failed silently. No model truncated the 32k prompt, every model finished the tool chain in exactly five calls, and none leaked reasoning text. The only gate-1 miss was speed, and it was a known trade-off of dense models, not a fault.

Two measurement details made the numbers trustworthy:

  • Unique-token prompts. A repeated prompt measures the server’s prefix cache, not the model. Random unique words force a real prefill every time.
  • Checking the echoed token count. Silent truncation is the most dangerous serving failure, because every later score still looks plausible. Comparing the token count the server reports with the prompt we sent catches it.

What this means if you serve models on a Spark

  1. Pick an MoE model for anything interactive. With 3–4B active parameters they decode at 52–66 tok/s and use about 28 GB.
  2. Treat a dense 27B as a batch model. At about 20 tok/s it’s fine for overnight extraction or screening, and slow for chat or agent loops.
  3. Probe speculative decoding on your real workload. The same draft head gave 39% acceptance on prose and 87–100% on tool calls.
  4. Run a needle test and check prompt_tokens before trusting any long-context result.
  5. Budget memory per model, not per box. Even the 55.7 GB Coder-Next leaves room on 121 GB for other services.

Limits. These are single-machine probes with the rest of the box idle, at one quantisation per model and one llama.cpp build. Load times were recorded but are not comparable across models (Gemma 4 was already resident from a warm-up). Qwen3-Coder-Next was tested after the L2 batch because its download finished mid-phase. Speed on real tasks varies with prompt length and output shape; see each task post for measured throughput.