Write the Evals First

In conventional software, you'd never ship a change without tests. With LLM features, teams routinely do exactly that: tweak a prompt, try three examples by hand, and merge. Because the output is probabilistic, it feels like testing doesn't apply. It applies more.

An evaluation set, or eval, is a collection of inputs paired with a way to judge the output. It's how you turn "seems better" into "is better."

Start with fifty real examples

You don't need thousands. Fifty carefully chosen cases from real usage beat a thousand synthetic ones. Include:

• The common case, several times over
• Edge cases you've already seen go wrong
• Inputs where the right answer is "I can't help with that"
• Adversarial inputs: prompt injection attempts, off-topic requests, abusive text

Pick the cheapest reliable grader

Not every check needs a model to grade it.

• Code checks for anything deterministic: valid JSON, required fields present, length limits, banned phrases.
• Reference comparisons when there's a known right answer, such as a classification label or an extracted date.
• Model-graded rubrics for qualities like tone, completeness, or faithfulness to a source. Write the rubric as specifically as you'd brief a new hire, and spot-check the grader against human judgment.

Run it on every change

Wire the eval into your pipeline so it runs whenever a prompt, model, or retrieval setting changes, and track scores over time. A drop in one category after a fix in another is the most common regression in LLM systems. The eval is the only thing that will catch it before your users do.

Feed production back in

When a user reports a bad answer, it becomes a test case. Over months, the eval set grows into a record of every way the feature has failed, and a guarantee that it won't fail that way again.

Read the failures, not just the score

A single overall score hides more than it shows. A change that lifts the average can still break the cases your most valuable customers depend on. Break results down by category, and read a sample of failing outputs after every run. Ten minutes spent reading failures usually tells you more about what to fix next than the headline number.

Version the eval like code

Keep the eval set in the repository, review changes to it in pull requests, and never quietly delete a case because it's failing. When the expected answer genuinely changes, say because a policy or a product rule changed, update the case and note why. An eval that's edited to pass stops being an eval.

If you can't say how you'd measure an improvement, you're not ready to make one.

Writing evals first feels slow for about a week. After that, it's the reason a team can change models, cut costs, or add capabilities with confidence instead of fear.