AI-native QA insights for modern teams.
Release discipline, test strategy, and AI workflow design — opinionated writing for teams that want stronger quality systems, not more noise.
What Is a Golden Dataset for Evals?
To evaluate whether AI is any good, you first need a set of human-verified "correct examples" as a standard — that's a golden dataset. This piece covers what it is, how to build a useful one, and why it goes stale like test cases do.
How to Monitor AI Behavior After Launch
An AI feature can change on its own without you changing anything — real user inputs and vendor model updates both do it. This piece covers what signals to monitor post-launch and how to turn real-traffic problems into test cases.
How to Test For and Reduce AI Hallucinations
The most dangerous thing about AI hallucination is being confidently wrong. This piece covers systematically testing for it — ground-truth questions, trap questions, faithfulness checks — and reducing it in design via RAG, citations, and allowing "I don't know."
LLM-as-a-Judge: When Can You Trust AI Evaluating AI?
Using another LLM as a judge to auto-score solves the scale problem, but the judge has biases too. This piece covers when LLM-as-a-Judge is trustworthy, when it's only a first pass, and how to calibrate the judge's accuracy.
How to Test a RAG Application: Verifying Retrieval-Augmented Generation
A RAG answer chains retrieval and generation, so when it's wrong you first have to tell which step failed. This piece covers verifying RAG quality across three layers — retrieval, generation, end-to-end — and why a knowledge base update means re-testing.
One Word in a Prompt, and Everything Changed: A Regression Testing Strategy for AI Products
Traditional regression testing relies on re-running the same assertions; AI products face a far less stable target. This piece covers building an AI regression suite, and why "different" doesn't mean "worse."
How Engineering Leads Can Pitch a Test Management Tool to Leadership
Engineering leads know they need a new tool, but struggle to translate engineers' pain into language a manager understands. This piece gives a practical pitch structure: 3 angles, 5 components, and common mistakes to avoid.
Red-Teaming Your Own AI Agent: Adversarial Testing Before Launch
Behavior evals verify an AI behaves correctly under normal use; red-teaming verifies whether it can be provoked into doing something it shouldn't. This piece covers 4 attack angles worth testing and why results still need human judgment.
Smoke Test vs Sanity Test vs Regression Test, Explained
Smoke, sanity, and regression testing get used interchangeably, but they answer three different questions. This piece breaks down scope, timing, and how the three fit into an actual test cycle.
No QA at Your Startup? A Minimum Viable Testing Process for Engineers
No dedicated QA isn't the problem — no testing process is. This piece gives you a 5-step minimum viable process that needs no extra tooling and no QA hire — start today.
What Is a Traceability Matrix?
What is a traceability matrix, what does it look like, and why do teams build one? This piece covers forward vs. backward traceability, and the most common built-in limitation of a spreadsheet-based matrix.
OpenTestX vs Xray: Which One Fits Your Team?
An honest comparison of OpenTestX and Xray: Xray's native Jira integration, how its pricing model actually works, and the different problem OpenTestX focuses on — active tracking when requirements change.