2,360 Offers, 40 Categories: Small Models Land Two Points Short on Product Data
Marketplaces and price-comparison sites take in millions of product offers, each written by a different shop in its own style. Before an offer is useful, someone has to file it under the right category in the catalogue and pull out a few key fields, like the brand and the barcode.
The volume is huge and each offer is worth very little, so this is where a local model’s cost advantage should matter most. Task E3 asks whether a model on one desk-side box can do the filing as accurately as a large model, at a fraction of the price.
Almost. Gemma 4 26B-A4B put 96.0% of offers in the right category and Qwen3-Coder-Next 95.5%. Both clear the 95% bar, but Claude Opus 5.5 scored 98.1%, and the rule allows a 2-point gap. Gemma misses it by 0.08 points.
Every model returned valid JSON on every call, none invented a category, and the local cost is under a cent per thousand offers. About 1.6 points of the gap sit in two pairs of easily confused labels.
The task in one picture
One call per offer. The model picks from a fixed list; it can’t make up a category because the list is built into the output format.
Enforcing the list uses constrained decoding: the server only lets the model emit text that fits the schema. DeepSeek’s API doesn’t offer that, so it got the list in its prompt and the scorer checked its answers instead.
The data
The offers come from WDC-222, a gold standard from Web Data Commons at the University of Mannheim. Each offer is a real web-shop listing linked to its product record in Icecat, an open product catalogue, and that record supplies the answers.
| Field | What it is | Scored on |
|---|---|---|
| Category | Icecat leaf category, one of 40 | all 2,360 offers (the primary metric) |
| Product type | The parent group, one of 18 | all 2,360 offers |
| Brand | As written in the offer | 1,454 offers where it appears in the text |
| GTIN flag | Does the text contain a valid barcode number? | all offers; 8 contain one |
A GTIN is the number under a barcode (EAN and UPC codes are kinds of GTIN). We kept the 40 most frequent categories, which gives 2,360 offers.
The comparison is a classic text classifier: TF-IDF word weights plus logistic regression, trained on 88,501 Icecat records and tested on the web offers. That is the setup the dataset’s own paper uses.
What a call looks like
You enrich e-commerce product offers. For each offer return JSON with: - category: the single best Icecat leaf category from this list: [...] - product_type: the parent group of that category, from: [...] - brand: the brand in lowercase as written, or null - gtin_in_text: true only if the offer text itself contains a GTIN/EAN/UPC barcode number (8, 12, 13 or 14 digits)
Title: msi gs73vr 7rf 207uk nvidia gtx 1060 6gb gddr5 17 3 uhd ips 4k
intel i7 7700hq gaming laptop
Description: gs73vr 7rf stealth pro 4k 207uk 17 3 uhd ips 4k 3840 x 2160 i7
7700hq nvidia gtx 1060 6gb gddr5 256gb ssd 2tb 7200rpm hdd ...{"category": "Notebooks", "product_type": "Computers", "brand": "msi",
"gtin_in_text": false}{"category": "Notebooks", "product_type": "Computers", "brand": "msi",
"gtin_in_text": false}The two offers below are where the models split. Offer A is “dymo s0929120 multi purpose labels 25mm x 25mm labelwriter label”; offer B is “sony mdrex450 inear headphones with aluminium housing and mic black”.
| Model | Offer A (DYMO labels) | Offer B (Sony in-ear) |
|---|---|---|
| Gold | Printer Labels | Mobile Headsets |
| Opus 5.5 | ✓ Printer Labels | ✓ Mobile Headsets |
| DeepSeek | ✓ Printer Labels | ✕ Headphones |
| Gemma 4 | ✕ Label-Making Tapes | ✕ Headphones |
| Qwen3-Coder-Next | ✓ Printer Labels | ✕ Headphones |
| Qwen3.8-27B | ✕ Label-Making Tapes | ✕ Headphones |
| Nemotron 3.5 | ✕ Label Printers | ✕ Headphones |
For offer A, the DYMO product is a roll of die-cut labels for a LabelWriter printer, so “Printer Labels” is right. Gemma and Qwen3.8 picked the tape-cassette category, and Nemotron picked the printer itself.
For offer B, the text says “headphones”, but Icecat files in-ear sets with a microphone as Mobile Headsets. Only Opus followed that convention here.
Results
The pass rule was fixed before any model ran: at least 95% category accuracy with no invented categories, and within 2 points of Opus. Local models ran three times at their recommended settings, and the headline is the median of the three runs.
Offers filed in the correct category
2,360 offers, 40 categories. The axis starts at 85%. Run-to-run spread is at most 0.8 points for every model.
| Model | Category | Product type | Brand | GTIN flag (false alarms) | Gap to Opus | Verdict |
|---|---|---|---|---|---|---|
| Opus 5.5 | 98.05% | 93.8% | 99.7% | 99.9% (0.0%) | – | baseline |
| DeepSeek | 97.80% | 94.8% | 99.5% | 99.8% (0.2%) | −0.25 | reference |
| Gemma 4 | 95.97% | 85.3% | 99.5% | 99.4% (0.5%) | −2.08 | needs a large model |
| Coder-Next | 95.51% | 91.6% | 99.0% | 99.4% (0.6%) | −2.54 | needs a large model |
| Qwen3.8 | 94.70% | 84.0% | 99.6% | 99.9% (0.0%) | −3.35 | needs a large model |
| Qwen3.6 | 94.19% | 91.3% | 99.5% | 99.6% (0.3%) | −3.86 | needs a large model |
| Nemotron 3.5 | 89.58% | 82.1% | 95.0% | 92.3% (7.7%) | −8.47 | needs a large model |
| TF-IDF classifier | 87.54% | – | – | – | −10.51 | – |
Brand extraction and the barcode flag are close to solved for everyone but Nemotron. Against DeepSeek, the cheaper API reference, Gemma is 1.83 points behind, which would be inside a 2-point gap.
The simple classifier is worth a second look. Overall it trails at 87.5%, but on the 49% of offers where it is at least 90% confident, it is 98.8% accurate.
Where the errors come from
We tallied every category error by its (gold, predicted) pair and read the largest groups against the offer text.
- A third of the errors are hard rows. 41 offers are missed by both large references. About a third of each local model’s errors fall in that set (Gemma 31 of 95, Coder-Next 34 of 104).
- Headsets versus headphones is a convention. Every model and both references file some in-ear sets with a mic as Headphones, 7 to 18 times each. The gold follows Icecat’s rule, so everyone is scored wrong for a defensible answer.
- Labels versus tapes is a real mistake. Die-cut LabelWriter labels filed as Label-Making Tapes: Gemma 29 times, Qwen3.8 30, Coder-Next 14, while the references make the reverse error 9 times. This is the main thing separating Gemma from the references.
- Nemotron has errors of its own. It labels the consumable as the device (Printer Labels as Label Printers, 67 times), files monitors as TVs (38), and flags barcodes that aren’t there on 7.7% of offers.
- Leaf and parent don’t always agree. Gemma and Qwen3.8 get the parent product type right less often (84–85%) than the leaf category, so they sometimes pick a pair that doesn’t fit together.
The scorer itself checked out: the allowed values matched the taxonomy, and every rejected answer was a real, in-list wrong label.
What this means if you run a product catalogue
The gap is small and concentrated, which makes it cheap to attack.
- Add a one-line rule per confusable pair to the prompt (“die-cut labels for a label printer are Printer Labels”). That’s where about 1.6 points of the gap sit.
- Derive the parent type from the leaf in code instead of asking the model for both. It removes the leaf/parent mismatches for free.
- Split by confidence. Let the TF-IDF classifier take the half of offers it’s sure about (98.8% accurate there) and send the rest to the local model.
- Use Gemma 4 or Qwen3-Coder-Next. Both clear the absolute bar; Gemma is the most consistent, with 99.1% agreement between its runs.
- If you need parity today, DeepSeek-V4.1-Flash is within 0.25 points of Opus at $0.52 per thousand offers.
A small fine-tune is another route, but the LoRA post shows why it has to be trained on the right data.
Limits. Only the 40 most frequent categories; long-tail categories will be harder. The gold follows Icecat's conventions, some of which differ from what the offer text says. References ran once each and local models three times.
