Log Triage: Severity Is Solved, Fault Labels Are a Convention
Operations teams spend their days reading log streams. For each line they ask three things: is this a fault, how urgent is it, and what actually happened in the last hour?
It is high-volume, low-margin text work, and logs often can’t leave the site. That makes it a natural fit for a local model, whose cost per line is close to zero. Task E2 tests the three parts separately: label a log line by fault type, rate an alert’s severity, and write an hourly summary for whoever is on call.
Narrowly, for the fault labels. Qwen3.8-27B reaches a macro-F1 of 0.861 (explained below), above the 0.85 bar, but 5.7 points behind Claude Opus 5.5 (0.918) where the rule allowed 3. It is within 1.5 points of DeepSeek-V4.1-Flash.
Severity is solved locally. Gemma 4, Qwen3.8 and Qwen3.6 rate 98.8–100% of alerts exactly and miss no critical one. Nearly all of the fault-label gap is lines that the human experts left untagged but that read like faults.
The task in one picture
Three subtasks share one pipeline shape: numbered input, constrained JSON out, scored by code or by a judge model.
Macro-F1 needs a word. For each of the five types, F1 balances “of the lines labelled X, how many really are X” against “of the real X lines, how many were found”. Macro-F1 is the plain average over the five types, so the rare types count as much as the common ones. 1.0 is perfect.
The data
The log lines come from Loghub (Zenodo 8196385, CC BY 4.0), a public collection of real system logs. We used two supercomputer logs, BGL and Thunderbird, where experts tagged the lines that are alerts.
| Subtask | Data | Gold |
|---|---|---|
| Fault type | 1,000 Loghub lines: 600 expert-tagged alerts (150 each of hardware, kernel, network, application) and 400 untagged lines, 200 per system | Tag → type map, frozen before any run; untagged = noise |
| Severity | 500 synthetic alerts | Rule-based severity 1–4 |
| Summary | 20 one-hour BGL windows with alerts, up to 120 lines each | A 1–5 score from a judge model |
The judge is DeepSeek. Opus judged Opus’s own summaries before its API credit ran out, and the two judges agreed within 1 point on all 20. DeepSeek may favour its own summaries, so the reference models’ summary scores are shown but are not used as a bar.
What a call looks like
Example 1: four lines, one convention
The prompt defines each type, for example noise: informational, routine or self-corrected events that need no action. Here are four lines (shortened):
RAS KERNEL FATAL Power deactivated: R01-M1-NCbn971 kernel: [KERNEL_IB][tsIbTavorMadProcess] ...RAS KERNEL FATAL program interrupt: fp compare....RAS KERNEL INFO CE sym 35, at 0x171288c0, mask 0x08
And what each model called them; ✕ marks a label that differs from the gold:
| Line 1 | Line 2 | Line 3 | Line 4 | |
|---|---|---|---|---|
| Gold | hardware | network | noise (untagged) | noise (untagged) |
| Opus 5.5 | hardware | network | ✕ kernel | noise |
| DeepSeek | hardware | network | ✕ kernel | noise |
| Qwen3.8 | hardware | network | ✕ kernel | noise |
| Gemma 4 | hardware | network | ✕ kernel | noise |
| Qwen3.6 | hardware | network | ✕ kernel | ✕ hardware |
| Nemotron 3.5 | hardware | network | ✕ kernel | ✕ kernel |
Line 3 is the hard case. The line says FATAL, and every model, both references included, calls it a kernel fault. The experts simply didn’t tag it, so the gold’s “noise” is a labelling convention, not something a reader could infer. Line 4 is where models genuinely differ: a corrected memory error (“CE”), which the prompt’s rule makes noise. Four models follow the rule; Qwen3.6 and Nemotron don’t.
Example 2: one hour of log, three summaries
The hour (BGL, 2005-08-22 23:00, 63 lines) holds one real incident: a fatal machine check on node R50-M0-N5-C:J15-U01 at 23:43, with a d-cache flush parity error in the status dump. Around it, a node on rack R37 logs corrected cache parity errors, and there are scattered corrected errors elsewhere.
"At 23:43, compute node R50-M0-N5-C:J15-U01 took a fatal machine check: a d-cache flush parity error in user state. ... Pull that node card for diagnostics ... R37-M1-NC-C:J02-U11 logged 12 corrected instruction-cache parity errors in 34 minutes. Monitor it ..."
Accurate, correct severity, actionable.
"A fatal hardware failure occurred on node R50-M0-N5-C:J15-U01 ... caused by a d-cache flush parity error ... Action: Inspect and replace the failed node and monitor ..."
Overstates the link to a shutdown: that line is from another node.
"... a fatal machine check was recorded at rack R50 ... The error includes ... I-cache parity error, and multiple DDR ECC correctable errors ... Immediate service action required."
Reports an I-cache parity error (the log shows i-cache 0, d-cache flush 1) and pins other racks' errors on this node.
Severity calls look like this: the alert gpu12 kernel: EXT4-fs error on /data: No space left on device; disk usage 100%, writes failing is a 4 (critical)
under the rubric, and every model answered 4.
Results
The pass rules, fixed in advance: fault-type macro-F1 at least 0.85 and within 3 points of Opus; severity at least 90% exact, within 3 points, and no severity-4 alert missed. A summary mean of 4/5 was reported but not part of the decision.
Fault-type macro-F1 (×100)
1,000 Loghub lines, 5 types. Local models are the median of 3 sampled runs.
| Model | Type F1 | Δ Opus | Alert lines right | Severity exact | Sev-4 missed | Summary (mean, share ≥ 4) | Verdict |
|---|---|---|---|---|---|---|---|
| Opus 5.5 | 0.918 | – | 96.5% | 100% | 0 | 4.95, 100% | baseline |
| DeepSeek-V4.1-Flash | 0.875 | −4.3 | 99.2% | 100% | 0 | 4.85, 100% | reference |
| Qwen3.8-27B | 0.861 | −5.7 | 97.7% | 98.8% | 0 | 3.90, 70% | needs a large model |
| Gemma 4 26B-A4B | 0.813 | −10.5 | 97.0% | 100% | 0 | 4.20, 90% | needs a large model |
| Qwen3.6-35B-A3B | 0.802 | −11.6 | 94.5% | 99.4% | 0 | 3.70, 60% | needs a large model |
| Qwen3-Coder-Next | 0.711 | −20.7 | 89.0% | 88.0% | 0 | 3.30, 40% | needs a large model |
| Nemotron 3.5 30B-A3B | 0.579 | −33.9 | 72.8% | 78.2% | 7 | 2.85, 25% | needs a large model |
The overall verdict is the worse of the type and severity parts. On severity alone, Gemma 4, Qwen3.8 and Qwen3.6 are small model ready; Qwen3-Coder-Next and Nemotron 3.5 are not. “Sev-4 missed” is the worst of the runs. Run-to-run spread is small here: Qwen3.8’s three runs were 0.861, 0.873 and 0.861, and its greedy run 0.880.
The gap is the noise row
The alert lines are nearly solved. On the 600 expert-tagged lines Qwen3.8 (97.7%) and Gemma 4 (97.0%) are level with Opus (96.5%), and DeepSeek reaches 99.2%. Almost nothing lands off the diagonal for the four fault types.
The whole difference is in the 400 untagged lines. Here is how many of them each model kept as noise:
Untagged lines kept as noise
400 lines the Loghub experts did not tag. No bar applies; this is the main driver of the F1 gap.
Two different things are going on in those lines:
- Lines that read like faults. 27 untagged BGL lines say FATAL, such as
RAS KERNEL FATAL program interruptregister dumps. Every model, the references included, gets most of them “wrong”. The references miss 66 lines together, and those make up 42% of Qwen3.8’s errors. - Lines the prompt’s rule covers. Opus wins mostly on
RAS KERNEL INFOlines, such as corrected-errorCE sym …reports, which the prompt says are noise. Opus keeps 100% of those as noise, Qwen3.8 66% and Qwen3.6 39%. Gemma calls 142 of the 400 untagged lines “kernel”.
So the task is partly a labelling convention. A site that wants the experts' alert set can add a short list of its benign patterns to the prompt. That is a prompt fix, not a model gap.
What the summaries got wrong
We read the judge’s reason for every summary scored 3 or below, 43 in all. The failure is rarely a missed incident. It is an invented detail sitting next to a correct one:
- unsupported causal chains, such as “R37 cache errors escalated into the R50 fatal”;
- invented actions or states, such as “routine maintenance replacement” or “R37 is offline”;
- wrong node ids, such as R67-M1-N3 for a fatal on R67-M1-N9;
- “no immediate action” next to fatal events.
Low scores per model: Nemotron 15 summaries, Qwen3-Coder-Next 12, Qwen3.6 8, Qwen3.8 6, Gemma 4 2. On severity, Qwen3-Coder-Next under-rates borderline alerts, and Nemotron misses up to 7 critical alerts in a run, which alone rules it out for paging.
What this means if you run on-call
- Classify and rate locally. For the tagged alert stream and severity, Qwen3.8-27B or Gemma 4 is enough, at a fraction of a cent per thousand lines.
- Write down your noise. Put your site’s benign patterns (corrected errors, routine INFO lines) in the prompt. That is where the gap to Opus is.
- Use Gemma 4 for hourly summaries, the best local summariser at 4.2/5 with 90% of summaries at 4 or above, and still have a person glance at them. The typical error is a plausible detail that isn’t in the log.
- Don’t page on Nemotron 3.5. Missing critical alerts is disqualifying.
Limits. The expert tags define the gold, and an untagged line is not benign in every case. The severity gold is synthetic and rule-based. The summary judge is DeepSeek, itself a reference model, and only 20 hours were summarised.
