#release-discipline
How we kept a red-team feature from being security theater
The hardest part of shipping a red-team feature isn't writing attacks — it's not faking them. Our merge gate: every attack bank must make a vulnerable agent go red and a hardened agent go green, or it fails the build. A probe that can't tell a broken agent from a fixed one is theater. This is the gate, and the three real bugs it caught by going red first.
Red-teaming an agent, without writing a single attack: a walkthrough
We added promptfoo's red-team thinking to Eval Studio by fusing it into the eval spine instead of bolting it on — our leak-veto was already an attack grade. This post is the concrete walkthrough: pick attack plugins from a library, run them against your agent, and read a matrix that says 'held against this probe' — never 'safe'. Built and merged, not yet deployed.
An access-control eval that actually arrives as the user
An access-control test is worthless unless the request arrives as the role being tested. So in Eval Studio each role carries a real credential — a pasted revocable token, or one minted against your own endpoint — and a cross-role leak-veto runs as an engine default, not an opt-in. This post walks through setting one up. Built and merged, not yet deployed.
One run, many axes — and no winner we can't defend
A Sweep varies model, prompt, language, and tenant over one suite and lays them out in a single matrix. The hard part isn't the cartesian — it's refusing the green 'winner' that's really just noise. This post walks through running a Sweep: pick your axes, read the cost gate, and read a matrix that calls a tie a tie. Built and merged, not yet deployed.
Evals for the QA who owns quality but doesn't write YAML
More teams are shipping agentic features than know how to test them — and the one tool that can, promptfoo, makes you write evals as code. Eval Studio brings that capability to the person whose job is quality, not the person who happens to write Python. This is the first post in a series on how we built it.
Our risk pill almost shipped a lie. Our own eval caught it.
We built a Gantt board that flags its own slipping groups and drafts a catch-up plan from your team's methodology. Then our own A/B eval told us the proof was rigged — so we fixed the product, not the test. The honest 5 → 1 → 5.
The And-step you write becomes a runnable automation step
A single-line Given/When/Then can't carry a real multi-step scenario. We added a verbatim Gherkin field and made every write path — import, chat, manual, CSV round-trip — preserve your newlines byte-for-byte, so the steps you author become the steps your code runs.
Reviewing dozens of pending cases without losing your place
Approving AI-generated test cases used to mean a modal that opened, then closed, then dropped you back at the top of the list. We rebuilt review as a persistent three-zone workspace where context never resets.
Coverage you can act on: the gap is the button
OpenTestX rebuilt its Coverage tab around an authoritative requirement catalog. A requirement with zero tests now shows as a gap card you generate tests from in one click — the gap is the button.