AI-native QA insights for modern teams.
Release discipline, test strategy, and AI workflow design — opinionated writing for teams that want stronger quality systems, not more noise.
Evals for the QA who owns quality but doesn't write YAML
More teams are shipping agentic features than know how to test them — and the one tool that can, promptfoo, makes you write evals as code. Eval Studio brings that capability to the person whose job is quality, not the person who happens to write Python. This is the first post in a series on how we built it.
Our risk pill almost shipped a lie. Our own eval caught it.
We built a Gantt board that flags its own slipping groups and drafts a catch-up plan from your team's methodology. Then our own A/B eval told us the proof was rigged — so we fixed the product, not the test. The honest 5 → 1 → 5.
Our coral sat 9° from our error red. We fixed color by measuring, not feeling.
A user told us twice the colors were too similar. The first fix over-corrected — every category chip went gray, and uniform is not the same as clear. The second fix put a number on it: our brand coral lives at hue 9°, our error red at 0°. Nine degrees apart, same lightness. You don't argue with 9°. You move.
The governance flywheel: when recurring review findings write their own methodology rule
Every AI review finding already lands in a per-case worklist. But the patterns across cases were going unread. The flywheel reads them — and when a weakness has truly recurred, it proposes one grounded SKILL.md rule into the queue you already approve from. The SQL decides. The model only phrases. A human still signs.
Drop a case onto a plan — the target appears only while you drag
Grab a test case from the Library tree, drag it down, and a drop strip materializes at the bottom of the screen — listing only the plans that can actually accept the case. Zero idle pixels, no closed ledgers reopened by accident.
The AI rewrites your Gherkin — one click brings the original back
Every content edit to a test case now snapshots a version, with a server-computed diff. So when the AI rewrites a scenario to fix a review finding, the byte-exact original is one click away. AI you can audit and undo.
AI review findings that stop vanishing
One click runs an AI review on a pending test case. The findings don't evaporate when you look away — they persist as a worklist with a real lifecycle, each one actionable, and they quietly feed everything downstream.
The And-step you write becomes a runnable automation step
A single-line Given/When/Then can't carry a real multi-step scenario. We added a verbatim Gherkin field and made every write path — import, chat, manual, CSV round-trip — preserve your newlines byte-for-byte, so the steps you author become the steps your code runs.
Reviewing dozens of pending cases without losing your place
Approving AI-generated test cases used to mean a modal that opened, then closed, then dropped you back at the top of the list. We rebuilt review as a persistent three-zone workspace where context never resets.
Coverage you can act on: the gap is the button
OpenTestX rebuilt its Coverage tab around an authoritative requirement catalog. A requirement with zero tests now shows as a gap card you generate tests from in one click — the gap is the button.