Write the evals before the prompt: testing LLM features properly

Every LLM project we have rescued had the same missing piece. There was a prompt, sometimes a very good one, and there was a demo that impressed the right people. What there was not was a way to tell whether Tuesday's change made things better or worse. Teams were editing prompts based on the last complaint they heard, fixing one case and quietly breaking three others. The cure is unglamorous: write the evaluation set before you write the prompt, and treat it as the specification for the feature.

Why evals come first

Traditional software has unit tests because code is deterministic and behavior can be asserted. LLM features are probabilistic, which tempts teams to skip testing altogether. That gets the logic backwards. Because output varies, you need more measurement, not less, and you need it before you have an opinion about which prompt is best.

Writing evals first forces three conversations that otherwise happen too late:

• What does correct mean? For a summarizer, is a missing detail a failure or a style issue? For a classifier, which confusions are costly and which are harmless?
• What must never happen? Leaking another customer's data, inventing a price, or promising a refund are different from being slightly unhelpful, and they deserve separate tracking.
• What is good enough to ship? Agreeing on a threshold, say 90% correct with zero critical failures, before seeing results stops the goalposts from moving.

Building a useful evaluation set

The best source of test cases is real historical data: past tickets, emails, documents or queries that the feature would have handled. We aim for 150 to 300 cases for a first release. Fewer than 100 and the noise between runs swamps real differences; more than 500 and the set becomes expensive to maintain before you know what matters.

Our process for a new feature looks like this:

1. Sample real inputs, stratified so rare but important categories are represented, not just the common ones.
2. Have a domain expert write or approve a reference outcome for each case. This is the slowest step and the most valuable.
3. Tag each case with category and severity so results can be sliced later.
4. Add a small adversarial set: prompt injections, off-topic requests, abusive messages and inputs in unexpected languages.
5. Freeze a holdout of 20% that nobody looks at while iterating, to catch overfitting to the visible cases.

That last step catches more self-deception than any other. We have seen prompts reach 94% on the working set and 81% on the holdout, because the engineer had unconsciously tuned wording to specific examples.

Graders you can trust

Grading is where evals usually fall apart. Where possible, use deterministic checks: did the classifier return the right label, did the agent call the right tool with the right arguments, does the extracted date match the reference. These checks are cheap, fast and unambiguous.

For open-ended output such as summaries or replies, a model-based grader with a written rubric is practical, but only after calibration. We have two people grade the same 60 outputs independently, measure their agreement, then tune the rubric and grader prompt until the model agrees with the humans at least as often as they agree with each other. Typically that lands between 85% and 92%. Without calibration, you are measuring the grader's taste, not the feature's quality.

An uncalibrated model grader is just a second prompt you have not tested.

Keep rubrics specific. "Is this answer helpful" produces noise. "Does the answer state the correct cancellation deadline, cite the policy page, and avoid promising a refund" produces signal.

Wiring evals into delivery

Evals only change behavior when they run automatically. We add the suite to CI so that any change to prompts, retrieval settings, tool definitions or model version triggers a run and posts a comparison to the pull request. A typical gate blocks the merge if overall correctness drops by more than one point or if any critical-severity case regresses.

A few practical details make this sustainable:

• Cache model responses keyed by input and configuration, so re-running unchanged cases costs nothing.
• Run each case three times for features where variance is high, and report the pass rate rather than a single outcome.
• Track cost and latency alongside quality. A prompt that gains two points but doubles latency is a product decision, not an automatic win.
• Store every run so you can chart quality over time and spot slow drift after model updates.

After launch, the evaluation set should grow from production. Sample a small percentage of live outputs for human review, and add every confirmed failure as a new case. Within a few months, the set reflects what actually goes wrong rather than what the team guessed might.

What this costs and what it saves

Building a first evaluation set usually takes four to six days of combined engineering and domain-expert time. Running it costs a few dollars per full pass for most features. Against that, we routinely see teams avoid shipping regressions that would have taken weeks of customer complaints to diagnose, and model upgrades become a one-afternoon decision instead of a leap of faith.

Every engagement in our LLM integration and custom AI agent practices starts with this work, and the evaluation suite is handed over with the code so your team can keep improving the feature after we leave. If you are also weighing whether to train a model, the same eval set is the only fair way to compare a tuned model against a well-prompted one, a question we explore in our model fine-tuning work.

Get your LLM feature measured

If you have an LLM feature in production and no reliable way to tell whether changes help, we can build the evaluation set and CI gate as a short, self-contained project. Describe the feature and its current pain points and we will reply within 24 hours with a fixed-price quote.