← Work

Evals for a model cascade — knowing when a prompt change makes things worse

nutri.'s AI extraction runs on a multi-model fallback chain. This is the golden-set eval harness that turns 'the prompt feels better' into a scored regression gate — including what it exposed about the weakest model in the chain.

nutri.’s AI route turns a food description or a label photo into a structured per-100 g nutrition estimate, riding a multi-model fallback chain (now gemini-3.6-flash → gemini-3.5-flash → gemini-2.5-flash → qwen3.6-27b on Groq → mistral-small-latest). Before this harness existed, changing a prompt, a schema or a model was an act of faith: the output looked fine. This case is the eval harness that replaced faith with a scored regression gate — and what building it exposed.

Ground truth is a discipline, not a fixture

The golden set is seven cases, each with expected values sourced from USDA FoodData Central (SR Legacy / Foundation), fdcId recorded in the case file — never from a model’s memory, which is exactly the contamination an eval exists to catch. Label cases carry a full hand-transcription of the package panel. The photos themselves stay local-only (the repo is public; kitchen photos aren’t), and the runner skips a case cleanly when its photo is missing, so any clone still runs green.

Transcribing is not estimating

One scoring function, two tolerance regimes. Reading a printed label is transcription: max(2%, 0.5 u). Estimating a raw food is inference: ±30% on macros, ±40% on micros. Seven mandatory fields — the four macros plus sodium, potassium and magnesium — must come back numeric on every run, tolerance or not. And a seed case that fails against FDC does not get “fixed” by loosening its tolerance: that failure is model-quality signal, so it’s documented and the baseline captures the real state.

Count hallucinations, don’t name them

On complete label transcriptions, any micro the model returns beyond what the package declares is a fabrication. The catch: which micros get invented varies run to run — the identity is noise, the count is the stable signal. So the gate tracks the extra-field count and fails when it grows beyond 1.5× + 3 versus baseline.

Determinism is something you build

The production cascade is an availability feature and an eval poison: one 503 and a different model answers, and a re-run flags regressions that are really just gemini-2.5 disagreeing with gemini-3.5. The runner pins a single model at temperature 0, retries transient 5xx errors, and never retries a 429 — quota is not a transient. Even pinned, an estimation field at the tolerance edge can wobble; the policy is one deliberate re-run, and if the failure persists, it’s real.

One baseline per model — and what that exposed

The default baseline gates the primary model. A second, separately-committed baseline covers the chain’s last resort, mistral-small-latest — which turned out to be weak and non-reproducible even at temperature 0: it hallucinates most of the micro panel, reads a label’s “per serving” column while ignoring the declared values, and returned 47 then 37 kcal for the same photo on consecutive runs. The honest response was to downgrade that baseline to a smoke test — is the Mistral route alive, does the Gemini→Mistral schema translation still hold — instead of a blocking gate. An eval whose false alarms you learn to ignore is worse than no eval.

The committed baselines make the gap concrete. Fields counts expected values that came back within tolerance; invented counts micros returned beyond what the package declares (meaningful on label cases, where anything extra is a fabrication — the gate tracks growth against baseline, not zero):

The Gemini Flash models return essentially every expected field within tolerance; the last-resort model fabricates most of the micro panel on every label it reads. That table is the argument for the smoke-test demotion.

Update: the harness earning its keep on launch day

Google shipped Gemini 3.6 Flash on 2026-07-21. The next morning, promoting it to the head of the chain took one pinned eval run (seven requests) against its own fresh baseline — no faith required. The numbers made the call: invented micros across the label cases dropped from 85 with 3.5-flash to 21, with the worst offender (huevo-gallinita, 70 fabricated micros) going to zero. The run also surfaced the trade-offs a “looks better” vibe check would have missed: one case (miel-aguacate) got worse on fabrications (8 → 19), and 3.6 omits an exempt-but-required field (potassium on a label that doesn’t declare it) instead of returning null. Net favorable, shipped the same session — with the regressions documented in the committed baseline instead of discovered by a user. This is what the eval was built for: the day a new model launches, “should we switch?” is a 15-minute measurement, not a debate.

Update: onboarding a fourth provider in one afternoon

On 2026-07-22 the chain grew a free-tier redundancy link: qwen3.6-27b on Groq, slotted between the Gemini models and Mistral. The eval decided the model, not the docs — and the docs would have lied twice. The model every 2026 guide recommends for vision on Groq, Llama 4 Scout, turned out to be gone from the live API catalog (404); the only vision-capable model actually served was qwen3.6-27b. And that model’s first real run failed every case with an opaque json_validate_failed — it was spending its entire output budget on hidden reasoning and returning empty JSON, fixed by forcing reasoning_effort: "none", which is the right call anyway for deterministic transcription work.

The harness needed to learn one genuinely new lesson: a 429 is not a 429. Gemini’s means daily quota — retrying is throwing requests into a dead window, so the runner treats it as fatal. Groq’s means a per-minute token window (8k TPM measured from the response headers) — fully retryable thirty seconds later. Same status code, opposite handling, now encoded per provider.

The fresh baseline over the same seven cases: 59/72 fields, with exactly one invented micro across all four label cases — the lowest fabrication count of any model in the chain, below even gemini-3.6-flash. Its weakness is the mirror image of Mistral’s: it doesn’t invent, it under-delivers, systematically missing the mandatory potassium/magnesium estimates on labels that don’t declare them. For a fourth-choice fallback that only ever answers when three Gemini models are down, that trade — honest gaps over confident fabrications — is exactly the one to want, and the committed baseline documents it instead of a hunch.

Update: when the gate fires, arbitrate with an A/B — not a re-run

Later on 2026-07-22, a three-line accuracy change (temperature 0 in production to match the eval, Gemini’s MEDIA_RESOLUTION_HIGH for sharper label OCR, and label photos compressed to 2048 px instead of 1024) tripped the hallucination gate: miel-aguacate jumped from 19 invented micros to 66. The change looked guilty. A controlled A/B on that exact case — same photo, same prompt, with and without the new media setting — returned zero fabrications on both arms. The gate had caught model variance, not the change: even pinned at temperature 0, vision output isn’t fully deterministic, and the invented-micro count itself can swing 0 → 66 on one case across runs. That sharpens the earlier lesson — the count is more stable than the identity, but it still wobbles — and it settled the arbitration policy: when the count gate fires on a change that shouldn’t plausibly cause it, a one-variable A/B on the failing case costs two requests and gives an answer; blind re-runs cost seven and burn the day’s quota proving nothing.

Cost is part of the spec

The free tier allows 20 requests per day per model; a run costs seven. That constraint shaped the tool: the eval is a deliberate before/after instrument, never a CI step (quota, plus non-determinism), while the scoring logic is pure, network-free and tested in CI on every push.

The PM takeaway: “how would you eval this feature?” stops being abstract once you’ve had to decide what counts as ground truth, why transcription and estimation deserve different tolerances, how to gate on hallucinations that never repeat themselves, and when an eval should honestly demote itself to a smoke test. The harness lives in the open repo: github.com/vryahn/nutri, evals/.