One run, many axes — and no winner we can't defend

This is the second post in our Eval Studio series. The first was about who evals are for — the QA who owns quality but doesn't write YAML. This one is about the thing that QA actually stares at: the matrix, and what happens when you want to vary more than one thing at once.

When you ship an agent, "does it work" is never one question. It's: does it work on this model and that one, with the old prompt and the new one, in English and in Japanese, for an admin and for a read-only user. Each of those is an axis. The naive answer is a full cartesian — run every combination — and the naive cartesian invites a specific lie. The whole point of this post is the part where we refuse to tell it.

We call the unified thing a Sweep: one suite, varied across any combination of axes, rendered as one matrix. Four axes generalize for free because every axis rides the same engine — model and prompt become the target columns, language becomes sibling input rows, and tenant/role becomes the context partition. The matrix was already the right surface; we just made it multi-dimensional without making it dishonest.

How you run a Sweep

Here is the happy path, in order. A reader following these steps lands on the same matrix.

• 1. Start a Sweep and pick the base suite. A Sweep is one suite — its inputs, its assertions, its expectations — varied across axes. Everything that follows holds that suite fixed; only the axes move.

• 2. Tick the axes to vary. Model and prompt (the target columns), language (sibling input rows), tenant/role (the context partition). Add the variants for each axis you tick — two models, three prompts, two languages. You're describing a grid, and you can see it forming.

• 3. Read the cost gate before you enqueue. The single most important screen in the flow. Below.

• 4. Confirm and run. The runner fans out the cells, grades each one, and assembles the matrix.

• 5. Read the result via the slice-pivot — and respect the tie-band. One swept axis becomes the columns; a second becomes a slicer above the grid; language renders as independent sub-bands. The tie-band tells you when a difference is too small to act on.

The cost gate: bounded before you spend

A cartesian multiplies, and the bill multiplies with it. Four models × three prompts × three languages is thirty-six runs per behavior — on someone's API key. So before a Sweep enqueues a single call, you get a gate: the cell count, and what it will cost.

• For known providers — gemini, OpenAI, Anthropic, with published per-token pricing — the gate shows a bounded dollar range, computed from the cell count, an estimated input size, and a conservative output cap. A range, labeled an estimate, never a single confident number we can't stand behind.

• For a bring-your-own endpoint — your own model, opaque pricing — the gate shows the call count and "cost unknown". We will not fabricate a dollar figure for an endpoint whose pricing we can't see. Honest blank beats a confident lie.

• And there's a hard ceiling. A Sweep that would exceed MAX_SWEEP_CELLS (500 cells) is refused, not silently truncated. Three or more axes that explode past the ceiling is the engine's signal to run two Sweeps, not one runaway grid.

The cost gate, before any call is made. The multiplier is spelled out deterministically — '2 behaviors × 2 inputs × 2 targets = 4 graded cells' — with the calls on your own key, a 'cost unknown' note for bring-your-own endpoints, and the 500-cell ceiling shown. Cancel here and nothing is enqueued.

How to read the matrix (the behavior view)

A Sweep result is a grid of behaviors against target columns, with a verdict in each cell and a drill-down into the exact input, output, and how it was graded. This is the behavior matrix — a verdict-and-divergence read, distinct from the held/breached attack matrix in the red-team post later in this series. Two things make it trustworthy rather than merely colorful.

Language is a sub-band, never a winner

Here is the construction decision that matters most. A cross-language winner is impossible in a Sweep — by construction, not by policy. Languages aren't aggregated into a score; an English phrasing and a Japanese phrasing are different inputs, each graded alone, rendered as independent sub-bands. The verdict logic ranks target columns and has no aggregate-winner field that could span languages. So the matrix can never tell you "the Japanese prompt wins" — because that sentence isn't expressible. Every serious multilingual benchmark is human-translated for exactly this reason: translation noise contaminates a fair comparison, and it does the most damage precisely on the single per-cell verdict a QA screenshots. We refuse to average it away.

A tie is called a tie

The second guard is the one we're proudest of, because we caught ourselves breaking it. An early version of our own verdict code stamped a winner from a raw pass-count — 11 out of 12 beats 10 out of 12, declare a winner. That is noise wearing a green badge, and worse, it had two unit tests that were themselves false-greens, quietly asserting the wrong thing. We replaced it with a Wilson-confidence-interval tie-band: when two targets' intervals overlap, the matrix says ≈ within noise — a tie — not a winner. An 11/12-vs-10/12 difference is not a shipping decision, and the matrix now says so out loud.

The slice-pivot: target columns across the top, behavior rows down the side, and language as independent sub-bands (BASELINE / JA), each graded alone. There is nowhere in this grid for a cross-language winner to appear, because the data model has no such field.

The Wilson tie-band chip: '≈ within noise'. When two targets' confidence intervals overlap — here 0% (range 0–79%) versus 100% (range 21–100%), the ranges shown on hover — the result is a tie, not a winner. This chip is the difference between a comparison and a coin-flip dressed as a decision.

Why a prompt change can't masquerade as a winner

One more integrity boundary, because it's subtle and it's where comparisons usually go wrong. "Comparable-by-construction" only holds for axes that don't change what's being scored. Model and tenant are safe — they're configuration around a fixed suite. Prompt and language are dangerous: a "prompt variant" that secretly changes the task would produce a green "prompt B wins" on a real shipping decision, when prompt B simply got an easier job. So we model a prompt variant as a same-model target with a different request template — a configuration-plane change, with the suite's scoring and expectations held identical — and two things stand guard: a config-plane guard that enforces model-equality across a prompt comparison, plus a falsifier test that goes red if a content-plane change ever renders as a winner, so the build knows. The comparison is allowed to be honest because the engine refuses to let it be otherwise.

What's true today — and what isn't

The honest scope, the same as every post in this series. The Sweep is built and merged, and we've verified the multi-axis path end-to-end in a real browser and against our own eval harness — the cost gate, the slice-pivot, the language sub-bands, the tie-band, the falsification guard. It is not deployed; this is a release-discipline post about work that's done and waiting at the gate. Two honest limits worth naming: the tie-band uses a Wilson-overlap test, and the proper next-pass method is a paired bootstrap — the v1 band is honest, not final. And we cap a Sweep at 500 cells rather than pretend an unbounded queued cartesian is free. When we say a comparison holds, we mean we watched the matrix refuse to lie — not that it's live for you to run yet.