Most AI projects have the same shape. A clever prompt, a model selection, a working demo, a sprint of iteration on prompts, a launch. Then six weeks pass, the model gets cheaper, someone swaps it out, and nobody can prove the new behaviour is better — or worse.
The fix is not exotic. It is to treat evaluation as a deliverable, not an afterthought. We will not ship a system without:
- A golden set of inputs and expected outputs, owned by the people who care about the answers.
- A scoring function that runs in CI on every prompt change, model swap, or retrieval tweak.
- A regression alert when score drops below the previous baseline, with a one-click way to view the offending traces.
This is not glamorous work. It is also non-negotiable. A system you cannot measure is a system you cannot ship — at least not with a straight face.
The shape of a useful eval
The best eval sets are small (200–500 examples), opinionated (your actual edge cases), and adversarial (the things that broke last quarter). They are not synthetic. They are not "scraped from the web." They are the receipts of the team's own pain.
Building one takes a week. Skipping that week costs a quarter.
What "passing" means
Passing is not 100%. Passing is known. If the system gets 87% of the golden set right, and the failure mode is documented, you can ship. If it gets 99% right but you cannot tell us how the 1% fails, you cannot.
A system can ship at 70% golden-set accuracy — as long as the other 30% routes to a human, with the model's draft attached. That is a perfectly good system. The thing that makes it good is that everyone knows exactly where the 70% ends.
The takeaway
If your AI roadmap does not have a line item for building the eval set, the rest of the roadmap is a guess.