<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom">
  <channel>
    <title>LLM Evaluation on 67AI Lab</title>
    <link>https://67ailab.com/tags/llm-evaluation/</link>
    <description>Recent content in LLM Evaluation on 67AI Lab</description>
    <generator>Hugo</generator>
    <language>en-us</language>
    <lastBuildDate>Thu, 12 Nov 2026 14:50:00 +0000</lastBuildDate>
    <atom:link href="https://67ailab.com/tags/llm-evaluation/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Which Jobs Can a Small Local Model Take Over? Results from Eight Real Tasks</title>
      <link>https://67ailab.com/posts/spark-eval-01-overall-results/</link>
      <pubDate>Thu, 12 Nov 2026 14:50:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-01-overall-results/</guid>
      <description>&lt;p&gt;Most teams that use a frontier model through an API are paying frontier&#xA;prices for work that isn&amp;rsquo;t frontier work: sorting, extracting, filling in&#xA;forms, following a script. The obvious question is how much of that a small&#xA;open model could do instead.&lt;/p&gt;&#xA;&lt;p&gt;The usual answers don&amp;rsquo;t help much. Leaderboards measure someone else&amp;rsquo;s tasks,&#xA;and a model that is &amp;ldquo;within a few points on average&amp;rdquo; can still fail in&#xA;exactly the way that hurts you. So we measured it on &lt;strong&gt;eight narrow,&#xA;realistic tasks&lt;/strong&gt;, with a pass rule written down for each one before any&#xA;model ran.&lt;/p&gt;</description>
    </item>
    <item>
      <title>62 GPU-Hours on One Box: How We Ran a Small-Model Evaluation on a DGX Spark</title>
      <link>https://67ailab.com/posts/spark-eval-02-evaluation-process/</link>
      <pubDate>Thu, 12 Nov 2026 14:40:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-02-evaluation-process/</guid>
      <description>&lt;p&gt;The other parts of this series ask what small models can do. This one is&#xA;about how we found out: what it takes to run five open models through eight&#xA;real tasks, two API references and a fine-tune, on one desk-side machine.&lt;/p&gt;&#xA;&lt;p&gt;That machine is an NVIDIA DGX Spark: a GB10 Grace Blackwell chip with&#xA;&lt;strong&gt;121 GB of unified memory&lt;/strong&gt;, meaning the CPU and GPU share one pool, on an&#xA;aarch64 (ARM) system. Unified memory is the Spark&amp;rsquo;s appeal, since a 35B model&#xA;fits easily. It also means every process on the box draws from the same pool:&#xA;the model, the test containers, your other services, even the agent driving&#xA;the study.&lt;/p&gt;</description>
    </item>
    <item>
      <title>Before Any Benchmark: Can Five Open Models Run Well on One DGX Spark?</title>
      <link>https://67ailab.com/posts/spark-eval-03-serving-smoke-test/</link>
      <pubDate>Thu, 12 Nov 2026 14:30:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-03-serving-smoke-test/</guid>
      <description>&lt;p&gt;An evaluation is only as good as the serving setup under it. If the server&#xA;quietly cuts the front off a long prompt, or a tool call comes back malformed,&#xA;every later score still looks plausible, and every later score is wrong.&lt;/p&gt;&#xA;&lt;p&gt;So before spending GPU days on eight real tasks, we asked a plainer question:&#xA;can each candidate model be served on one NVIDIA DGX Spark, at a usable speed,&#xA;without silent failures? This is task T01, the study&amp;rsquo;s first gate. A model that&#xA;failed here would not be worth testing further.&lt;/p&gt;</description>
    </item>
    <item>
      <title>Can a Local Model Screen a Systematic Review? Yes — Here&#39;s the Recipe</title>
      <link>https://67ailab.com/posts/spark-eval-04-l1-literature-screening/</link>
      <pubDate>Thu, 12 Nov 2026 14:20:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-04-l1-literature-screening/</guid>
      <description>&lt;p&gt;Screening is the slowest part of a systematic review. A search returns a&#xA;few thousand papers, and two people read every title and abstract to find the&#xA;2–3% that meet the review&amp;rsquo;s criteria.&lt;/p&gt;&#xA;&lt;p&gt;A screening assistant doesn&amp;rsquo;t have to make that call. It only has to &lt;strong&gt;rank&lt;/strong&gt;&#xA;the papers, so that people who read the top half still see nearly every paper&#xA;that matters. That&amp;rsquo;s the question for task L1: can a model running on a&#xA;desk-side box rank well enough to halve the reading without losing papers?&lt;/p&gt;</description>
    </item>
    <item>
      <title>Turning Scan Reports into JSON Is Solved. Spotting Contradictions Isn&#39;t</title>
      <link>https://67ailab.com/posts/spark-eval-05-l2-report-structuring/</link>
      <pubDate>Thu, 12 Nov 2026 14:10:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-05-l2-report-structuring/</guid>
      <description>&lt;p&gt;Imaging departments hold years of free-text scan reports. Registries, audits&#xA;and research projects need the same reports as structured fields: which&#xA;tracer, how much activity, how long after injection, and for each lesion its&#xA;site, side, size and uptake.&lt;/p&gt;&#xA;&lt;p&gt;Doing that by hand is slow, and the reports often can&amp;rsquo;t leave the hospital&#xA;network. Task L2 asks whether a model on a desk-side box can do the conversion&#xA;unattended, and whether it notices when a report contradicts itself.&lt;/p&gt;</description>
    </item>
    <item>
      <title>Rarely Wrong, Often Incomplete: Small Models Answering from Clinical Guidelines</title>
      <link>https://67ailab.com/posts/spark-eval-06-l3-guideline-qa/</link>
      <pubDate>Thu, 12 Nov 2026 14:00:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-06-l3-guideline-qa/</guid>
      <description>&lt;p&gt;Technologists and physicians in a nuclear-medicine department look up dosing,&#xA;patient preparation and safety rules in procedure guidelines many times a day.&#xA;An assistant that answers from those guidelines would save time, and running&#xA;it locally would keep every question inside the hospital.&lt;/p&gt;&#xA;&lt;p&gt;The bar is strict, though. An answer must carry every key fact, because the&#xA;caveat attached to a number is often the safety part. The assistant must say&#xA;so when the guideline doesn&amp;rsquo;t cover a question. And a confident wrong dose&#xA;rules a model out. Task L3 asks whether any local model clears that bar.&lt;/p&gt;</description>
    </item>
    <item>
      <title>When the Answer Key Is the Ceiling: Pulling Numbers out of Trial Papers</title>
      <link>https://67ailab.com/posts/spark-eval-07-l4-trial-extraction/</link>
      <pubDate>Thu, 12 Nov 2026 13:50:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-07-l4-trial-extraction/</guid>
      <description>&lt;p&gt;A meta-analysis pools the results of many randomised trials into one estimate.&#xA;To do that it needs a few numbers from every paper: how many patients were in&#xA;each arm, and either how many had the event (for a yes/no outcome) or the mean&#xA;and standard deviation (for a measured one).&lt;/p&gt;&#xA;&lt;p&gt;Pulling those numbers out by hand is slow, fiddly work. They hide in tables,&#xA;footnotes and results text, and standard practice is to extract everything&#xA;twice, by two people, and reconcile. A model that fills the extraction sheet&#xA;for a person to check would save much of that time. Task L4 asks whether a&#xA;model on a desk-side box can do it well enough.&lt;/p&gt;</description>
    </item>
    <item>
      <title>Fixing Real Bugs in Real Repos: Where Small Models Still Trail by 22.5 Points</title>
      <link>https://67ailab.com/posts/spark-eval-08-e1-scoped-coding/</link>
      <pubDate>Thu, 12 Nov 2026 13:40:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-08-e1-scoped-coding/</guid>
      <description>&lt;p&gt;A coding agent is the hardest everyday job you can give a language model. It&#xA;has to read a bug report, find its way around a repository it has never seen,&#xA;edit the right file, run the tests and then stop. Each step depends on the&#xA;last, so one bad move early on can sink the whole attempt.&lt;/p&gt;&#xA;&lt;p&gt;It&amp;rsquo;s also the job teams most want to bring in-house, because code is the data&#xA;they least like sending to an API. Task E1 asks a narrow question: can a model&#xA;running on one desk-side box close well-described issues in real open-source&#xA;projects often enough to be useful?&lt;/p&gt;</description>
    </item>
    <item>
      <title>Log Triage: Severity Is Solved, Fault Labels Are a Convention</title>
      <link>https://67ailab.com/posts/spark-eval-09-e2-log-triage/</link>
      <pubDate>Thu, 12 Nov 2026 13:30:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-09-e2-log-triage/</guid>
      <description>&lt;p&gt;Operations teams spend their days reading log streams. For each line they ask&#xA;three things: is this a fault, how urgent is it, and what actually happened in&#xA;the last hour?&lt;/p&gt;&#xA;&lt;p&gt;It is high-volume, low-margin text work, and logs often can&amp;rsquo;t leave the site.&#xA;That makes it a natural fit for a local model, whose cost per line is close to&#xA;zero. Task E2 tests the three parts separately: label a log line by fault type,&#xA;rate an alert&amp;rsquo;s severity, and write an hourly summary for whoever is on call.&lt;/p&gt;</description>
    </item>
    <item>
      <title>2,360 Offers, 40 Categories: Small Models Land Two Points Short on Product Data</title>
      <link>https://67ailab.com/posts/spark-eval-10-e3-product-enrichment/</link>
      <pubDate>Thu, 12 Nov 2026 13:20:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-10-e3-product-enrichment/</guid>
      <description>&lt;p&gt;Marketplaces and price-comparison sites take in millions of product offers,&#xA;each written by a different shop in its own style. Before an offer is&#xA;useful, someone has to file it under the right category in the catalogue and&#xA;pull out a few key fields, like the brand and the barcode.&lt;/p&gt;&#xA;&lt;p&gt;The volume is huge and each offer is worth very little, so this is where a&#xA;local model&amp;rsquo;s cost advantage should matter most. Task E3 asks whether a model&#xA;on one desk-side box can do the filing as accurately as a large model, at a&#xA;fraction of the price.&lt;/p&gt;</description>
    </item>
    <item>
      <title>A 3B-Active Model That Matches DeepSeek at Running Customer-Service Agents</title>
      <link>https://67ailab.com/posts/spark-eval-11-e4-tool-calling-agents/</link>
      <pubDate>Thu, 12 Nov 2026 13:10:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-11-e4-tool-calling-agents/</guid>
      <description>&lt;p&gt;A lot of business automation is a loop: talk to a customer, look up their&#xA;record, check the policy, change something in a database, and say what&#xA;happened. Booking changes, returns and phone-line troubleshooting all look&#xA;like this.&lt;/p&gt;&#xA;&lt;p&gt;An agent that does this needs more than fluent replies. It has to call the&#xA;right functions with the right arguments over many turns, follow written&#xA;rules even when the customer pushes back, and ask for missing details instead&#xA;of guessing. Task E4 asks whether a local model can do that as reliably as a&#xA;large model.&lt;/p&gt;</description>
    </item>
    <item>
      <title>The LoRA That Learned the Wrong Lesson: Fine-Tuning on Synthetic Papers</title>
      <link>https://67ailab.com/posts/spark-eval-12-lora-fine-tune/</link>
      <pubDate>Thu, 12 Nov 2026 13:00:00 +0000</pubDate>
      <guid>https://67ailab.com/posts/spark-eval-12-lora-fine-tune/</guid>
      <description>&lt;p&gt;Every task in this study ended in one of three verdicts: &lt;strong&gt;small model ready&lt;/strong&gt;,&#xA;&lt;strong&gt;needs a large model&lt;/strong&gt;, or &lt;strong&gt;needs tuning&lt;/strong&gt;. &amp;ldquo;Needs tuning&amp;rdquo; means a local model came within&#xA;the allowed gap of Claude Opus 5.5 but stayed under the absolute bar, so a&#xA;little training might push it over.&lt;/p&gt;&#xA;&lt;p&gt;Only one task landed there: &lt;a href=&#34;https://67ailab.com/posts/spark-eval-07-l4-trial-extraction/&#34;&gt;L4, pulling trial results out of papers for&#xA;meta-analysis&lt;/a&gt;. Qwen3.6-35B-A3B&#xA;scored 74.2% against Opus&amp;rsquo;s 75.8%, with a bar of 90%. So the last phase of the&#xA;study asked a simple question: can a small fine-tune, trained on the same box,&#xA;close that gap?&lt;/p&gt;</description>
    </item>
  </channel>
</rss>
