Small Models on One DGX Spark

Log Triage: Severity Is Solved, Fault Labels Are a Convention

Alert lines are 97% right for the best local models; untagged noise lines are where the gap sits

Operations teams spend their days reading log streams. For each line they ask three things: is this a fault, how urgent is it, and what actually happened in the last hour?

It is high-volume, low-margin text work, and logs often can’t leave the site. That makes it a natural fit for a local model, whose cost per line is close to zero. Task E2 tests the three parts separately: label a log line by fault type, rate an alert’s severity, and write an hourly summary for whoever is on call.

Needs a large model E2 verdict

Narrowly, for the fault labels. Qwen3.8-27B reaches a macro-F1 of 0.861 (explained below), above the 0.85 bar, but 5.7 points behind Claude Opus 5.5 (0.918) where the rule allowed 3. It is within 1.5 points of DeepSeek-V4.1-Flash.

Severity is solved locally. Gemma 4, Qwen3.8 and Qwen3.6 rate 98.8–100% of alerts exactly and miss no critical one. Nearly all of the fault-label gap is lines that the human experts left untagged but that read like faults.

The task in one picture

Three subtasks share one pipeline shape: numbered input, constrained JSON out, scored by code or by a judge model.

InputLines, alerts or an hourBatches of 20 numbered log lines; single synthetic alerts; or a whole hour of log
Local modelLabel, rate or summariseOne of five types per line; severity 1–4 per alert; a short on-call summary
ScoringCode, then a judgeLabels and severity against gold; summaries scored 1–5 by DeepSeek against the raw hour
MetricMacro-F1 and exact matchPlus: no severity-4 alert may be missed

Macro-F1 needs a word. For each of the five types, F1 balances “of the lines labelled X, how many really are X” against “of the real X lines, how many were found”. Macro-F1 is the plain average over the five types, so the rare types count as much as the common ones. 1.0 is perfect.

The data

The log lines come from Loghub (Zenodo 8196385, CC BY 4.0), a public collection of real system logs. We used two supercomputer logs, BGL and Thunderbird, where experts tagged the lines that are alerts.

Subtask Data Gold
Fault type 1,000 Loghub lines: 600 expert-tagged alerts (150 each of hardware, kernel, network, application) and 400 untagged lines, 200 per system Tag → type map, frozen before any run; untagged = noise
Severity 500 synthetic alerts Rule-based severity 1–4
Summary 20 one-hour BGL windows with alerts, up to 120 lines each A 1–5 score from a judge model

The judge is DeepSeek. Opus judged Opus’s own summaries before its API credit ran out, and the two judges agreed within 1 point on all 20. DeepSeek may favour its own summaries, so the reference models’ summary scores are shown but are not used as a bar.

What a call looks like

Example 1: four lines, one convention

The prompt defines each type, for example noise: informational, routine or self-corrected events that need no action. Here are four lines (shortened):

  1. RAS KERNEL FATAL Power deactivated: R01-M1-NC
  2. bn971 kernel: [KERNEL_IB][tsIbTavorMadProcess] ...
  3. RAS KERNEL FATAL program interrupt: fp compare....
  4. RAS KERNEL INFO CE sym 35, at 0x171288c0, mask 0x08

And what each model called them; ✕ marks a label that differs from the gold:

Line 1 Line 2 Line 3 Line 4
Gold hardware network noise (untagged) noise (untagged)
Opus 5.5 hardware network ✕ kernel noise
DeepSeek hardware network ✕ kernel noise
Qwen3.8 hardware network ✕ kernel noise
Gemma 4 hardware network ✕ kernel noise
Qwen3.6 hardware network ✕ kernel ✕ hardware
Nemotron 3.5 hardware network ✕ kernel ✕ kernel

Line 3 is the hard case. The line says FATAL, and every model, both references included, calls it a kernel fault. The experts simply didn’t tag it, so the gold’s “noise” is a labelling convention, not something a reader could infer. Line 4 is where models genuinely differ: a corrected memory error (“CE”), which the prompt’s rule makes noise. Four models follow the rule; Qwen3.6 and Nemotron don’t.

Example 2: one hour of log, three summaries

The hour (BGL, 2005-08-22 23:00, 63 lines) holds one real incident: a fatal machine check on node R50-M0-N5-C:J15-U01 at 23:43, with a d-cache flush parity error in the status dump. Around it, a node on rack R37 logs corrected cache parity errors, and there are scattered corrected errors elsewhere.

Opus 5.5 judge 5

"At 23:43, compute node R50-M0-N5-C:J15-U01 took a fatal machine check: a d-cache flush parity error in user state. ... Pull that node card for diagnostics ... R37-M1-NC-C:J02-U11 logged 12 corrected instruction-cache parity errors in 34 minutes. Monitor it ..."

Accurate, correct severity, actionable.

Gemma 4 judge 4

"A fatal hardware failure occurred on node R50-M0-N5-C:J15-U01 ... caused by a d-cache flush parity error ... Action: Inspect and replace the failed node and monitor ..."

Overstates the link to a shutdown: that line is from another node.

Nemotron 3.5 judge 2

"... a fatal machine check was recorded at rack R50 ... The error includes ... I-cache parity error, and multiple DDR ECC correctable errors ... Immediate service action required."

Reports an I-cache parity error (the log shows i-cache 0, d-cache flush 1) and pins other racks' errors on this node.

Severity calls look like this: the alert gpu12 kernel: EXT4-fs error on /data: No space left on device; disk usage 100%, writes failing is a 4 (critical) under the rubric, and every model answered 4.

Results

The pass rules, fixed in advance: fault-type macro-F1 at least 0.85 and within 3 points of Opus; severity at least 90% exact, within 3 points, and no severity-4 alert missed. A summary mean of 4/5 was reported but not part of the decision.

Fault-type macro-F1 (×100)

1,000 Loghub lines, 5 types. Local models are the median of 3 sampled runs.

Small modelLarge-model reference0.85 bar
Opus 5.50.918
DeepSeek V4.1 Flash0.875
Qwen3.8-27B0.861
Gemma 40.813
Qwen3.60.802
Qwen3-Coder-Next0.711
Nemotron 3.50.579
00.51.0
Model Type F1 Δ Opus Alert lines right Severity exact Sev-4 missed Summary (mean, share ≥ 4) Verdict
Opus 5.5 0.918 – 96.5% 100% 0 4.95, 100% baseline
DeepSeek-V4.1-Flash 0.875 −4.3 99.2% 100% 0 4.85, 100% reference
Qwen3.8-27B 0.861 −5.7 97.7% 98.8% 0 3.90, 70% needs a large model
Gemma 4 26B-A4B 0.813 −10.5 97.0% 100% 0 4.20, 90% needs a large model
Qwen3.6-35B-A3B 0.802 −11.6 94.5% 99.4% 0 3.70, 60% needs a large model
Qwen3-Coder-Next 0.711 −20.7 89.0% 88.0% 0 3.30, 40% needs a large model
Nemotron 3.5 30B-A3B 0.579 −33.9 72.8% 78.2% 7 2.85, 25% needs a large model

The overall verdict is the worse of the type and severity parts. On severity alone, Gemma 4, Qwen3.8 and Qwen3.6 are small model ready; Qwen3-Coder-Next and Nemotron 3.5 are not. “Sev-4 missed” is the worst of the runs. Run-to-run spread is small here: Qwen3.8’s three runs were 0.861, 0.873 and 0.861, and its greedy run 0.880.

$0.002–0.003per 1,000 lines locally, vs $0.87 on Opus and $0.26 on DeepSeek
≈ 3.1 hGPU time for 2.1k calls, plus the judge
0critical alerts missed by Gemma 4, Qwen3.8 or Qwen3.6

The gap is the noise row

The alert lines are nearly solved. On the 600 expert-tagged lines Qwen3.8 (97.7%) and Gemma 4 (97.0%) are level with Opus (96.5%), and DeepSeek reaches 99.2%. Almost nothing lands off the diagonal for the four fault types.

The whole difference is in the 400 untagged lines. Here is how many of them each model kept as noise:

Untagged lines kept as noise

400 lines the Loghub experts did not tag. No bar applies; this is the main driver of the F1 gap.

Small modelLarge-model reference
Opus 5.584.3%
DeepSeek V4.1 Flash66.7%
Qwen3.8-27B65.7%
Gemma 450.0%
Qwen3.650.0%
Qwen3-Coder-Next35.7%
Nemotron 3.527.8%
0%50%100%

Two different things are going on in those lines:

  • Lines that read like faults. 27 untagged BGL lines say FATAL, such as RAS KERNEL FATAL program interrupt register dumps. Every model, the references included, gets most of them “wrong”. The references miss 66 lines together, and those make up 42% of Qwen3.8’s errors.
  • Lines the prompt’s rule covers. Opus wins mostly on RAS KERNEL INFO lines, such as corrected-error CE sym … reports, which the prompt says are noise. Opus keeps 100% of those as noise, Qwen3.8 66% and Qwen3.6 39%. Gemma calls 142 of the 400 untagged lines “kernel”.

So the task is partly a labelling convention. A site that wants the experts' alert set can add a short list of its benign patterns to the prompt. That is a prompt fix, not a model gap.

What the summaries got wrong

We read the judge’s reason for every summary scored 3 or below, 43 in all. The failure is rarely a missed incident. It is an invented detail sitting next to a correct one:

  • unsupported causal chains, such as “R37 cache errors escalated into the R50 fatal”;
  • invented actions or states, such as “routine maintenance replacement” or “R37 is offline”;
  • wrong node ids, such as R67-M1-N3 for a fatal on R67-M1-N9;
  • “no immediate action” next to fatal events.

Low scores per model: Nemotron 15 summaries, Qwen3-Coder-Next 12, Qwen3.6 8, Qwen3.8 6, Gemma 4 2. On severity, Qwen3-Coder-Next under-rates borderline alerts, and Nemotron misses up to 7 critical alerts in a run, which alone rules it out for paging.

What this means if you run on-call

  1. Classify and rate locally. For the tagged alert stream and severity, Qwen3.8-27B or Gemma 4 is enough, at a fraction of a cent per thousand lines.
  2. Write down your noise. Put your site’s benign patterns (corrected errors, routine INFO lines) in the prompt. That is where the gap to Opus is.
  3. Use Gemma 4 for hourly summaries, the best local summariser at 4.2/5 with 90% of summaries at 4 or above, and still have a person glance at them. The typical error is a plausible detail that isn’t in the log.
  4. Don’t page on Nemotron 3.5. Missing critical alerts is disqualifying.

Limits. The expert tags define the gold, and an untagged line is not benign in every case. The severity gold is synthetic and rule-based. The summary judge is DeepSeek, itself a reference model, and only 20 hours were summarised.