Small Models on One DGX Spark

2,360 Offers, 40 Categories: Small Models Land Two Points Short on Product Data

96% of offers in the right category for Gemma 4, against 98.1% for Opus 5.5 and 97.8% for DeepSeek

Marketplaces and price-comparison sites take in millions of product offers, each written by a different shop in its own style. Before an offer is useful, someone has to file it under the right category in the catalogue and pull out a few key fields, like the brand and the barcode.

The volume is huge and each offer is worth very little, so this is where a local model’s cost advantage should matter most. Task E3 asks whether a model on one desk-side box can do the filing as accurately as a large model, at a fraction of the price.

Needs a large model, narrowly E3 verdict

Almost. Gemma 4 26B-A4B put 96.0% of offers in the right category and Qwen3-Coder-Next 95.5%. Both clear the 95% bar, but Claude Opus 5.5 scored 98.1%, and the rule allows a 2-point gap. Gemma misses it by 0.08 points.

Every model returned valid JSON on every call, none invented a category, and the local cost is under a cent per thousand offers. About 1.6 points of the gap sit in two pairs of easily confused labels.

The task in one picture

One call per offer. The model picks from a fixed list; it can’t make up a category because the list is built into the output format.

InputOne offerThe shop's title and description, often lower-case and run together
Local modelFill four fieldsCategory, product type, brand and a barcode flag, as JSON
CodeEnforce the listThe 40 categories and 18 types are allowed values in the JSON schema
MetricCategory accuracyShare of offers filed in the same category as the gold catalogue

Enforcing the list uses constrained decoding: the server only lets the model emit text that fits the schema. DeepSeek’s API doesn’t offer that, so it got the list in its prompt and the scorer checked its answers instead.

The data

The offers come from WDC-222, a gold standard from Web Data Commons at the University of Mannheim. Each offer is a real web-shop listing linked to its product record in Icecat, an open product catalogue, and that record supplies the answers.

Field What it is Scored on
Category Icecat leaf category, one of 40 all 2,360 offers (the primary metric)
Product type The parent group, one of 18 all 2,360 offers
Brand As written in the offer 1,454 offers where it appears in the text
GTIN flag Does the text contain a valid barcode number? all offers; 8 contain one

A GTIN is the number under a barcode (EAN and UPC codes are kinds of GTIN). We kept the 40 most frequent categories, which gives 2,360 offers.

The comparison is a classic text classifier: TF-IDF word weights plus logistic regression, trained on 88,501 Icecat records and tested on the web offers. That is the setup the dataset’s own paper uses.

What a call looks like

System prompt (excerpt)
You enrich e-commerce product offers. For each offer return JSON with:
- category: the single best Icecat leaf category from this list: [...]
- product_type: the parent group of that category, from: [...]
- brand: the brand in lowercase as written, or null
- gtin_in_text: true only if the offer text itself contains a
  GTIN/EAN/UPC barcode number (8, 12, 13 or 14 digits)
Input · wdc-6
Title: msi gs73vr 7rf 207uk nvidia gtx 1060 6gb gddr5 17 3 uhd ips 4k
       intel i7 7700hq gaming laptop
Description: gs73vr 7rf stealth pro 4k 207uk 17 3 uhd ips 4k 3840 x 2160 i7
       7700hq nvidia gtx 1060 6gb gddr5 256gb ssd 2tb 7200rpm hdd ...
Gold
{"category": "Notebooks", "product_type": "Computers", "brand": "msi",
 "gtin_in_text": false}
Gemma 4 all four correct
{"category": "Notebooks", "product_type": "Computers", "brand": "msi",
 "gtin_in_text": false}

The two offers below are where the models split. Offer A is “dymo s0929120 multi purpose labels 25mm x 25mm labelwriter label”; offer B is “sony mdrex450 inear headphones with aluminium housing and mic black”.

Model Offer A (DYMO labels) Offer B (Sony in-ear)
Gold Printer Labels Mobile Headsets
Opus 5.5 ✓ Printer Labels ✓ Mobile Headsets
DeepSeek ✓ Printer Labels ✕ Headphones
Gemma 4 ✕ Label-Making Tapes ✕ Headphones
Qwen3-Coder-Next ✓ Printer Labels ✕ Headphones
Qwen3.8-27B ✕ Label-Making Tapes ✕ Headphones
Nemotron 3.5 ✕ Label Printers ✕ Headphones

For offer A, the DYMO product is a roll of die-cut labels for a LabelWriter printer, so “Printer Labels” is right. Gemma and Qwen3.8 picked the tape-cassette category, and Nemotron picked the printer itself.

For offer B, the text says “headphones”, but Icecat files in-ear sets with a microphone as Mobile Headsets. Only Opus followed that convention here.

Results

The pass rule was fixed before any model ran: at least 95% category accuracy with no invented categories, and within 2 points of Opus. Local models ran three times at their recommended settings, and the headline is the median of the three runs.

Offers filed in the correct category

2,360 offers, 40 categories. The axis starts at 85%. Run-to-run spread is at most 0.8 points for every model.

Small modelLarge-model referenceTF-IDF classifier95% bar
Opus 5.598.05%
DeepSeek V4.1 Flash97.80%
Gemma 495.97%
Qwen3-Coder-Next95.51%
Qwen3.8-27B94.70%
Qwen3.694.19%
Nemotron 3.589.58%
TF-IDF classifier87.54%
85%90%95%100%
Model Category Product type Brand GTIN flag (false alarms) Gap to Opus Verdict
Opus 5.5 98.05% 93.8% 99.7% 99.9% (0.0%) – baseline
DeepSeek 97.80% 94.8% 99.5% 99.8% (0.2%) −0.25 reference
Gemma 4 95.97% 85.3% 99.5% 99.4% (0.5%) −2.08 needs a large model
Coder-Next 95.51% 91.6% 99.0% 99.4% (0.6%) −2.54 needs a large model
Qwen3.8 94.70% 84.0% 99.6% 99.9% (0.0%) −3.35 needs a large model
Qwen3.6 94.19% 91.3% 99.5% 99.6% (0.3%) −3.86 needs a large model
Nemotron 3.5 89.58% 82.1% 95.0% 92.3% (7.7%) −8.47 needs a large model
TF-IDF classifier 87.54% – – – −10.51 –

Brand extraction and the barcode flag are close to solved for everyone but Nemotron. Against DeepSeek, the cheaper API reference, Gemma is 1.83 points behind, which would be inside a 2-point gap.

The simple classifier is worth a second look. Overall it trails at 87.5%, but on the 49% of offers where it is at least 90% confident, it is 98.8% accurate.

0invented categories and 0 invalid JSON replies, across every model and run
$0.005–0.009per 1,000 offers locally, vs $8.16 on Opus and $0.52 on DeepSeek
≈ 8.6 hGPU time; about 52k calls in all

Where the errors come from

We tallied every category error by its (gold, predicted) pair and read the largest groups against the offer text.

  • A third of the errors are hard rows. 41 offers are missed by both large references. About a third of each local model’s errors fall in that set (Gemma 31 of 95, Coder-Next 34 of 104).
  • Headsets versus headphones is a convention. Every model and both references file some in-ear sets with a mic as Headphones, 7 to 18 times each. The gold follows Icecat’s rule, so everyone is scored wrong for a defensible answer.
  • Labels versus tapes is a real mistake. Die-cut LabelWriter labels filed as Label-Making Tapes: Gemma 29 times, Qwen3.8 30, Coder-Next 14, while the references make the reverse error 9 times. This is the main thing separating Gemma from the references.
  • Nemotron has errors of its own. It labels the consumable as the device (Printer Labels as Label Printers, 67 times), files monitors as TVs (38), and flags barcodes that aren’t there on 7.7% of offers.
  • Leaf and parent don’t always agree. Gemma and Qwen3.8 get the parent product type right less often (84–85%) than the leaf category, so they sometimes pick a pair that doesn’t fit together.

The scorer itself checked out: the allowed values matched the taxonomy, and every rejected answer was a real, in-list wrong label.

What this means if you run a product catalogue

The gap is small and concentrated, which makes it cheap to attack.

  1. Add a one-line rule per confusable pair to the prompt (“die-cut labels for a label printer are Printer Labels”). That’s where about 1.6 points of the gap sit.
  2. Derive the parent type from the leaf in code instead of asking the model for both. It removes the leaf/parent mismatches for free.
  3. Split by confidence. Let the TF-IDF classifier take the half of offers it’s sure about (98.8% accurate there) and send the rest to the local model.
  4. Use Gemma 4 or Qwen3-Coder-Next. Both clear the absolute bar; Gemma is the most consistent, with 99.1% agreement between its runs.
  5. If you need parity today, DeepSeek-V4.1-Flash is within 0.25 points of Opus at $0.52 per thousand offers.

A small fine-tune is another route, but the LoRA post shows why it has to be trained on the right data.

Limits. Only the 40 most frequent categories; long-tail categories will be harder. The gold follows Icecat's conventions, some of which differ from what the offer text says. References ran once each and local models three times.