An access-control eval that actually arrives as the user

This is the third post in our Eval Studio series, and the most security-flavored of them. The idea is simple to state and easy to get dangerously wrong: an access-control test is only meaningful if the request arrives as the role you're testing. If your eval asks "can a read-only user reach the admin endpoint?" but sends the request as nobody in particular, a green result proves nothing. So the first design decision was that a role in Eval Studio carries a real credential, and the request genuinely arrives as that role.

That single requirement has a trust problem baked into it, and how we resolved it is most of this post.

Paste or mint — never hand us a signing key

There are two ways to make a request arrive as a specific role. We could hold a signing secret and mint a token for any identity on the fly — convenient, and a non-starter. If we hold the key that signs your identities, we can forge any of them. That's a power we don't want and you shouldn't grant. We rejected it on trust grounds.

So the trustworthy floor we shipped first is paste: you hand over one scoped, revocable token per role — the kind you can mint yourself and kill in a dashboard the moment the eval is done. It's the most trustworthy shape available: we never hold a signing key, we hold a narrow token you control end to end. Paste was deliberately the foundation, because it makes the trust boundary obvious before any convenience is layered on.

Built alongside it — and now merged — is the form that keeps that exact property without the manual paste: instead of handing us a token, the engine mints a short-lived one at run time by calling your own token endpoint (OAuth2 client-credentials), so the secret stays on your side and we still never hold a signing key. It's fail-closed — a mint that can't reach your endpoint, or resolves to a blocked address, fails the run rather than falling through to something weaker — and the minted token is scrubbed the same way a pasted one is. Both mechanisms ship: paste the floor, mint the convenience, neither asking you for a signing key. Built and merged, not yet deployed.

Wherever a token goes, it's scrubbed: out of the stored raw fields, out of any judge prompt, out of error messages and the detail blob. A credential's job is to decide who the request arrives as — never to leak into storage or an LLM call.

How you set up a role-isolation eval

The happy path, in order.

• 1. Add a role to the role library and give it a credential. Use the Paste|Mint toggle: paste a masked, revocable token, or configure a mint against your own token endpoint so the engine fetches a short-lived one per run. Either way the credential is workspace-level — a role you define once is reusable across suites.

• 2. Add a sibling role. Isolation is a relationship between roles — typically a high-privilege role and a low-privilege one — so you need at least two. Paste the second token the same way.

• 3. Build or pick the suite and run it as those roles. The roles become the context partition of the matrix; each cell's request arrives carrying that role's real token.

• 4. Read the isolation matrix. A leaked sibling datum is a breach; an un-credentialed role is honestly not-provable. The vocabulary below.

The role library is workspace-level. Define a role and its credential once; reference it from any suite. Isolation needs at least two siblings — here an admin and a read-only role — to be a relationship worth testing.

The Paste|Mint toggle, Paste selected: a narrow, revocable token you control, masked and 'encrypted & never shown again'. Never a signing key that would let us forge any identity.

Mint mode, shipped alongside paste: instead of handing us a token, the engine mints a short-lived one against your own OAuth2 token endpoint at run time. The secret stays on your side; we still never hold a signing key.

The leak-veto is an engine default, not a checkbox

Here is the part we got wrong first, and the lesson worth the post. The safety spine of a role-isolation run is a cross-role leak-veto: for any cell run as role A, no other role's token (B, C, …) may appear in the response. A's own token echoing back is fine; a sibling's token surfacing means data crossed a boundary it shouldn't have.

The trap is making that veto something the author opts into. We did, at first — and a live combined-matrix run found the hole: the veto only armed where someone had authored a forbidden-token sentinel. A leak into a role that nobody had sentineled rendered as isolated — a false-green that the unit tests happily camouflaged, because the tests exercised the sentineled path. Building the UI first would have shipped a credential-leaker that looked clean.

The fix was to make the guarantee hold by construction. The engine now injects a synthetic cross-role veto for every cell that has sibling tokens — automatically, idempotently, with no authoring required. You can't forget to arm it, because there's no switch to forget. The lesson we wrote down: a security guarantee that depends on the author remembering to turn it on is not a guarantee — run it live, because passing unit tests do not mean the property holds.

How to read the isolation matrix

The isolation matrix uses the same held/breached/not-tested/inconclusive vocabulary as the red-team matrix later in this series (the leak-veto is, after all, an access-control attack grade — we explore that fusion in the red-team post). Three outcomes do the work here:

• Held — the request arrived as this role, and no sibling's data came back. The boundary held against this probe — not "the agent is secure", the same careful reading the whole series insists on.

• ⚠ Breached — a sibling role's datum showed up in the response. Data crossed a boundary. The cross-role veto fired, by construction, whether or not anyone authored a sentinel.

• ⃠ Not-provable — the role had no credential, so the request didn't actually arrive as that role. The cell still runs and still shows the agent's real reply — we never fake output — but it carries a not-tested stamp. This is the honesty hinge: an un-credentialed role is read as "isolation not tested", never as "isolated". A blank we can't fill is shown blank, not quietly painted green.

Three honest outcomes across the role columns: a held Pass, a breached Fail caught by the leak veto, and — the one that matters most — a not-provable, un-credentialed guest. That guest cell still runs and shows the agent's real reply, but is stamped 'isolation not tested', never 'isolated'.

Drilling the not-provable cell: the agent's real output is shown and even leak-checked — but because no credential made the request arrive as that role, it's stamped 'not tested', never read as isolated. It runs and shows real output; it is never quietly painted green.

What's true today — and what isn't

The honest scope, as in every post here. Role-isolation is built and merged, and we've verified the per-role-credential path end-to-end — in a real browser, against our own eval harness, and through a security review — including a live run that confirmed the cross-role veto catches a real leak into an un-sentineled role. It is not deployed. Both credential mechanisms ship: paste a scoped revocable token, or mint a short-lived one against your own endpoint (OAuth2 client-credentials, fail-closed, scrubbed) — paste was the trustworthy floor we built first, and minting is now merged alongside it. Neither asks you for a signing key. When we say a boundary held, we mean we watched the veto run by construction on a real credentialed request — not that your agent is certified isolated, and never that an un-credentialed cell proved anything at all.