Evaluating an LLM application in production means answering one question repeatedly: is it still doing the job well enough to keep shipping? A demo answers it once, by eye. A production system has to answer it on every prompt change, model upgrade and data refresh, and keep answering it after release. This guide covers the parts of an evaluation setup that hold up in practice: a test set, the right kind of grader for each check, human review, a regression gate in CI, and monitoring once real users arrive.
Why LLM evaluation is different from ordinary testing
Conventional tests compare output to an expected value. Language models rarely give the same answer twice, and many answers can be correct in different words. That is why pass-or-fail assertions alone fall short, a point we covered in QA in evaluating LLM output. Three other things make it harder:
- The system is more than the model. Prompts, retrieved context, tool calls and post-processing all change the result. Evaluate the application, not only the model.
- Things change underneath you. Hosted models are updated or retired, your documents change, and user behaviour drifts away from what you tested.
- Quality has several dimensions. An answer can be accurate but badly formatted, well written but unsupported by the sources, or correct but too slow and too expensive to serve.
Start by writing down what “good” means
Before choosing tools, list the qualities that matter for your product and how each one would be judged. For a support assistant that might be: answers grounded in the help centre, no invented policy, correct escalation when it cannot help, a consistent tone, and a response in under a few seconds. For an extraction feature it is mostly field-level accuracy and valid structure. Each quality should end up with a check, an owner and a threshold. If you cannot say how a quality would be measured, it will not be protected when someone changes the prompt.
Build an offline test set
The test set is the asset that makes everything else work. It is a versioned collection of inputs, with expected answers or grading notes where they exist. Good sources are:
- Real traffic. Anonymised questions from logs or a pilot are the most representative inputs you will get.
- Domain experts. The people who know the answers can write the hard cases and the reference answers.
- Known failures. Every bug report or bad answer becomes a test case, so it stays fixed.
- Edge and adversarial cases. Out-of-scope questions, ambiguous requests, very long inputs, other languages, and prompt-injection attempts.
- Synthetic cases. A model can generate variations to widen coverage, but review them; synthetic sets tend to be easier than real ones.
Tag each case by feature and risk, so results can be read per slice rather than as one average. A few hundred well-chosen cases usually tell you more than thousands of near-duplicates. Keep the set in version control next to the prompts, so a change and its evaluation are reviewed together.
Choose the right grader for each check
Not every check needs a model to grade it. Use the cheapest grader that measures the thing reliably.
| Grader | Good for | Cost and speed | Watch out for |
|---|---|---|---|
| Code-based checks | Valid JSON, schema, required fields, length, banned phrases, exact values | Free and instant | Only covers what can be stated as a rule |
| Reference comparison | Extraction and classification against a known answer | Cheap | Needs labelled answers; string similarity misses correct rewordings |
| LLM-as-judge | Groundedness, relevance, tone, following instructions | Moderate; one extra model call per check | Its own biases and drift; must be calibrated against people |
| Human review | Subtle correctness, domain judgement, bias, calibrating the judge | Slow and expensive | Inconsistent without a written rubric |
Using LLM-as-judge without fooling yourself
A second model grading the first is the only practical way to score qualities like groundedness at scale, but it is a measuring instrument and needs the same care as one. Known weaknesses include a preference for longer answers, sensitivity to the order in which options are shown, a tendency to favour output from its own model family, and scores that shift when the judge model is updated. To keep it honest:
- Ask narrow questions with a written rubric (“Is every claim in the answer supported by the context? Yes or no, then the unsupported claim”) rather than a 1–10 score for “quality”.
- Ask for the reason before the verdict, and store both, so failures can be read.
- For pairwise comparisons, run both orders and count only agreements.
- Calibrate: have people label a sample, compare with the judge, and fix the rubric until they agree on most cases. Repeat when you change the judge model.
- Pin the judge model version and keep it separate from the model being tested where you can.
Human review still has a job
People are needed to write the rubric, label calibration samples, review what the judge is unsure about, and look at a regular sample of production traffic. Use the same rubric for reviewers that the judge uses, and track how often reviewers agree with each other; if they do not agree, the rubric is the problem.
Make evaluation a regression gate in CI
Evaluation pays off when it runs automatically on every change that can affect output: prompts, model version, retrieval settings, tool definitions and post-processing code. A workable pattern:
- Run code-based checks on every pull request. They are fast and catch broken structure immediately.
- Run the full test set with judge-based checks on pull requests that touch prompts or model configuration, or nightly.
- Compare against the last accepted baseline, per slice, not only overall. A small overall gain can hide a large loss on one feature.
- Fail the build on agreed thresholds, for example any drop on the critical slice or any new failure on a known-bug case.
- Publish a report the reviewer can read: which cases changed, and why the grader thinks so.
Because output varies between runs, set thresholds with some margin, and for small sets run important cases more than once. Open-source tools such as promptfoo and DeepEval are built for this style of test-set-in-CI workflow, and tracing platforms such as LangSmith can store datasets and experiment results; a plain test runner with your own graders also works. This is the same idea as moving a manual regression suite into the delivery pipeline, which our quality engineering work does for conventional software.
Monitor quality, cost and latency in production
Offline tests cover what you thought of. Production shows you what you did not. Instrument each request so you can see the input, the retrieved context, the model and prompt version, tool calls, tokens, latency and the final output. OpenTelemetry has emerging conventions for generative AI spans if you want this in your existing observability stack. Then watch:
- Quality signals. Judge scores on a sample of live traffic, user feedback, retries and rephrased questions, and escalations to a person.
- Drift. New topics and question types that your test set does not cover. Sample them into the test set.
- Cost. Tokens per request and per user, by feature. A prompt change that doubles context length doubles the bill.
- Latency. Percentiles, not averages, including time to first token for streamed answers.
- Safety. Policy and personal-data checks on outputs, and flagged prompt-injection attempts.
Respect privacy when logging prompts: redact personal data, limit retention, and control who can read traces.
LLM evaluation checklist
- Quality criteria written down, each with a check, an owner and a threshold.
- A versioned test set built from real traffic, expert cases and known failures, tagged by feature and risk.
- Code-based checks for anything that can be stated as a rule.
- An LLM judge with a narrow rubric, a pinned version, and recorded agreement with human labels.
- Human review of a regular sample, using the same rubric.
- Evaluation running in CI on prompt, model and retrieval changes, compared against a baseline per slice.
- A model upgrade treated as a release: run the full set before switching.
- Tracing in production with prompt and model versions on every request.
- Dashboards and alerts for quality, cost, latency and safety.
- A loop that turns production failures into new test cases.
Where to start
If you have nothing today, start small: fifty real questions, a few code-based checks, one judge-based check for the quality that matters most, and a script that runs before every release. Grow it from production failures. Teams that do this can change prompts and switch models with confidence, because they can see what changed.
This is how we approach evaluation when we build AI products: evaluation and monitoring are part of delivery rather than added afterwards. You can see it in practice in testing the behavior of generative AI, and in Verity, the AI quality solution in our AI Studio. If your product answers from your own documents, the companion post on RAG in production covers the retrieval side.