The LoRA That Learned the Wrong Lesson: Fine-Tuning on Synthetic Papers
Every task in this study ended in one of three verdicts: small model ready, needs a large model, or needs tuning. “Needs tuning” means a local model came within the allowed gap of Claude Opus 5.5 but stayed under the absolute bar, so a little training might push it over.
Only one task landed there: L4, pulling trial results out of papers for meta-analysis. Qwen3.6-35B-A3B scored 74.2% against Opus’s 75.8%, with a bar of 90%. So the last phase of the study asked a simple question: can a small fine-tune, trained on the same box, close that gap?
No gain on the real task. The tuned Qwen3.6 scores 73.8% on L4 against 74.2% for the base model. The fine-tune is not adopted, and L4 stays at "needs tuning".
It did learn what it was trained on: 90.0% → 94.6% on held-out synthetic papers. On real papers it gets more reported values right, but it also fills in values the answer key leaves empty, and the two effects cancel out.
The task in one picture
L4’s dataset has no training split, and the study’s rule was that answer keys are built without seeing model outputs. So the training data had to be synthetic, built backwards from the answers: code picks the numbers first, and a paper is written around them.
A LoRA (low-rank adaptation) freezes the original model and trains a small set of extra weights that sit beside some of its layers. It is cheap to train and cheap to serve: the adapter here is 16.7 MB and loads at runtime on top of the same quantised model the study already used.
The data
The generator draws 2–3 arms, randomised and analysed group sizes (sometimes only one is reported), and 2–6 outcomes. Each outcome gets a reporting style:
| Outcome type | Reporting styles |
|---|---|
| Binary (events) | n (%), % only, events/N, not reported, figure only |
| Continuous (measured) | mean (SD), mean (SE), mean (95% CI), median (IQR), difference only, figure only |
The gold row follows L4’s conventions. Group size is the number analysed. An SD comes from an SE as SE·√n, or from a CI as √n·(hi − lo)/3.92. Medians, differences and figures give a null, because a meta-analysis can’t use them as a mean and SD.
| Item | Count |
|---|---|
| Papers written by DeepSeek | 1,451 |
| Passed the checks | 1,332 (7,610 outcome rows) |
| Held out as a synthetic dev set | 100 |
| Used for training | 1,232 |
The chats use the exact L4 format: the same system prompt, the paper plus numbered rows, and a JSON reply, with the row order shuffled.
What a training example looks like
A synthetic paper, SYN00041. Because it is generated, it can be shown in full.
arms: TORUL n=112, CULR n=386 (randomised = analysed) O1 continuous, style mean_ci: TORUL mean 300.4, 95% CI 259.4–341.4 CULR mean 409.5, 95% CI 404.1–414.9 O2 continuous, style mean_sd: TORUL 11.3 ± 1.5 s CULR 16.5 ± 5.6 s
At post-treatment, the Upper-Limb Motor Impairment Composite score was 300.4 (95% CI 259.4 to 341.4) with TORUL and 409.5 (95% CI 404.1 to 414.9) with CULR. The between-group difference favoured TORUL (p<0.001). ... Nine-Hole Peg Test completion time ... 11.3 ± 1.5 seconds with TORUL and 16.5 ± 5.6 seconds with CULR ...
{"intervention_group_size": 112, "comparator_group_size": 386,
"intervention_mean": 300.4, "intervention_standard_deviation": 221.38,
"comparator_mean": 409.5, "comparator_standard_deviation": 54.13}
SD from the CI: √112 × 82.0 / 3.92 = 221.38 and √386 × 10.8 / 3.92 = 54.13. The training loss is computed on this JSON reply only.
Compare it with the real L4 example. Here every requested number is either in the text or nulled by a clear rule. In the real paper, the hard part was deciding which of several numbers the annotators meant, or whether they left the cell empty. That difference turns out to be the whole story.
Training on the box
| Setting | Value |
|---|---|
| Base | Qwen/Qwen3.6-35B-A3B, bf16 (the full-precision weights of the served model) |
| Adapted layers | Full-attention q/k/v/o (10 of 40 layers) and the shared-expert MLP (all 40) |
| Size | r 16, alpha 32: 8.4 M trainable parameters (0.024%) |
| Schedule | 1 epoch, 77 steps, lr 1e-4 cosine, batch 1 × 16 accumulation |
| Sequence length | max 6,144 tokens, none truncated (median 2,206) |
| Loss | 0.074 → about 0.02 |
Two engineering details mattered. First, the checkpoint has to be loaded with the full multimodal model class and fed text only. The generic text-only loader picks a class whose layer names don’t match the checkpoint, which first showed up as “target modules not found”. A 3-step smoke run caught it.
Second, some layers were left alone on purpose. The linear-attention (Gated DeltaNet) projections were skipped because llama.cpp’s converter reorders their heads, and the routed experts because they are stored as fused 3-D tensors.
Results
The tuned model reran L4 under the same protocol as the others: 109 papers, 3 sampled runs (median reported) plus 1 greedy run, against the original answer key.
Base vs LoRA-tuned Qwen3.6-35B-A3B
Synthetic dev: 100 held-out papers, greedy. Real L4: 109 papers, 3,550 cells per run; the gold-known and gold-null groups pool all sampled runs.
| Model | Exact (median of 3) | Runs | Greedy | Gold-known | Ref-agreed | Invented numbers |
|---|---|---|---|---|---|---|
| Opus 5.5 | 75.8% | 1 run | – | 71.3% | 93.9% | 4.2% |
| DeepSeek-V4.1-Flash | 77.9% | 1 run | – | 75.2% | 96.5% | 4.0% |
| Qwen3.6-35B-A3B | 74.2% | 70.3 / 74.2 / 75.4 | 75.5% | 70.6% | 87.4% | 2.4% |
| Qwen3.6 + LoRA | 73.8% | 73.2 / 73.8 / 74.7 | 72.6% | 74.4% | 87.0% | 3.6% |
“Ref-agreed” drops the 685 cells (19%) where Opus and DeepSeek both disagree with the key. That is a wider cut than the 563 cells in the L4 post, which counted only cells where the two references also gave the same answer, so the reference-agreed numbers here are higher. Either way, the tuned model gains nothing on those cells. Its runs are tighter than the base model’s, but its greedy run is worse (72.6% against 75.5%).
Two effects that cancel
The headline hides two opposite changes. Split the cells by whether the answer key has a value, and they separate cleanly:
| Cell group (all sampled runs) | Base | Tuned |
|---|---|---|
| Key has a value: model is correct | 69.2% | 74.2% |
| Key has a value: model gives a value | 85.0% | 94.0% |
| Key is empty: model leaves it empty | 83.8% | 73.2% |
| Group sizes correct (key known) | 76–79% | 84–85% |
| Means and SDs correct (key known) | 53–60% | 57–63% |
| Event counts correct (key known) | 85–86% | 84–85% |
The LoRA made the model answer more often. That helped wherever an answer exists, mainly group sizes (+7 to +9 points) and means and SDs (+2 to +6). It hurt wherever the key is empty: the tuned model fills in 27% of those cells, against 16% for the base.
Why it didn’t transfer
The synthetic nulls were too easy. 14.7% of synthetic gold cells are null, and each comes from a clear rule: a median, a difference only, a figure, or a value not reported. Both models get every one of them right.
In the real answer key, 27.9% of cells are null, often by annotator convention. The text gives the value, or lets you compute it, but the annotators left it out. Training on papers where almost every requested number is present taught the model the wrong lesson: there is always a number to find.
The rest of the gap is the key itself. As the L4 audit found, on 16% of cells both references agree with each other and not with the key, and on 19% both differ from it. On the cells the references agree on, the base model is at 87% against 94–97% for the references, and the remaining real gap there is mostly mean and SD extraction.
What this means if you fine-tune on synthetic data
- Expect it to learn the synthetic distribution. It reliably did. Whether that transfers depends on how well the synthetic data copies the real labelling conventions, and here the key convention was when to leave a cell empty.
- Make the synthetic nulls look like real ones. The next thing worth trying, not run here: papers that report values only as change scores, adjusted estimates or subgroups, so the model learns when real annotators say “not extractable”. Or a small LoRA on human-adjudicated real papers kept apart from the test set.
- Score by cell group, not just the headline. Splitting gold-known from gold-null cells is what exposed the two opposite effects.
- Smoke-test the training setup first. A 3-step run caught a silent model class mismatch before four hours were spent.
- Serve adapters at runtime. A LoRA GGUF on the existing quantised base makes an A/B comparison a single line of server config.
For L4 itself, use the Qwen3.6 base model with greedy decoding and human review, or DeepSeek-V4.1-Flash.
Limits. One model, one adapter size and one epoch: only Qwen3.6 was tuned, to fit the time budget, and other layer choices or more data might behave differently. The synthetic papers were written by DeepSeek, one of the reference models. The real-task scores inherit every problem of the L4 answer key described in part 7.
