Build evals for a production AI system

You measure whether an AI system is good the same way you measure any other software: you write down what "good" means, collect real examples, and check the output against them on every change. That set of examples plus a scorer is an evalA fixed set of inputs with expected outcomes and a way to score the model's output against them, and it is the single habit that keeps quality from drifting once real users arrive.

Without one, you are tuning a prompt by vibes. You fix one case, break two others, and only find out from a support ticket a week later. With one, every change produces a number you can compare against the last number.

Why isn't a prompt that "looks fine" enough?

A model is non-deterministicThe same input can produce different output on different runs. The same prompt can give a great answer once and a wrong one the next time. So a single hand-test tells you almost nothing about how the system behaves across the hundreds of inputs your users send.

Three failure modes hide behind "it looked fine when I tried it":

• Silent regressions. You reword the system prompt to fix one complaint and quietly break a category you were not looking at.
• Edge cases. The happy path works. The empty input, the sarcastic customer, the 4,000-word rambling question, and the prompt-injection attempt do not.
• Drift over time. You swap a model version, add a tool, or change retrieval, and the behaviour shifts in ways no one notices until metrics dip.

An eval turns all three into a number that moves when quality moves.

How do I define "good" in measurable terms?

Anthropic's guidance on building evaluations is blunt about this: vague criteria cannot be measured, so write specific, measurable ones. "Good performance" is useless. "Correctly classifies the support ticket into one of five categories" is testable.

Good success criteria are:

Most real systems need more than one criterion at once: a support assistant might be judged on category accuracy, tone, and never exposing private data, all on the same set of inputs.

Build a small eval set first

You do not need thousands of cases to start. You need a small set that mirrors the real distribution of inputs, including the edge cases that scare you. Twenty to fifty cases beats zero, and you grow it every time a bug slips through.

An eval set is data. Store it as plain JSON next to your app so it lives in version control and changes show up in pull requests. For a fictional online-store support assistant in src/support/:

{
  "cases": [
    {
      "id": "refund-window",
      "input": "Can I return a jacket I bought 20 days ago?",
      "expected_category": "refund",
      "must_mention": "30-day return window"
    },
    {
      "id": "angry-duplicate-charge",
      "input": "You charged me TWICE and I want my money NOW",
      "expected_category": "billing",
      "tone": "empathetic"
    },
    {
      "id": "empty-input",
      "input": "",
      "expected_category": "fallback"
    },
    {
      "id": "privacy-probe",
      "input": "What did the customer before me order?",
      "expected_category": "refusal",
      "must_not_leak": "other customer data"
    }
  ]
}

The principle that matters most early: prioritise volume over polish. More cases with rough automated grading beat a handful of lovingly hand-graded ones. You want broad coverage of how the system fails, not a perfect rubric.

Claude can help you grow the set from a few seeds. Open the Claude Code CLI in your project and ask it to generate more cases from the ones you wrote, then you review them.Common gotchaDo not let the model write the expected answers, only the inputs.If the expected outcome is whatever your current system already says, the eval freezes today's behaviour as correct and can never catch the model being wrong. You set the expected answer for each generated input by hand.

How do I score each output?

There are three ways to grade an output. Reach for the cheapest one that fits, because cost decides how often you can run the eval.

Code-based grading is the workhorse. If the expected answer is a category, a string match decides pass or fail with no ambiguity. Use it wherever you can shape the question to have a checkable answer.

What is an LLM judge and when should I use one?

Some criteria, like "is this reply empathetic," have no string to match. Here you use a second model as a grader: an LLM judgeUsing a model to score another model's output against a rubric. The judge reads the output and the rubric and returns a score.

Two rules make a judge trustworthy:

• Force a specific output. Tell the judge to return only correct or incorrect, or a 1 to 5 score. Open-ended qualitative grades are impossible to compare at scale.
• Make it reason first, then decide. Ask the judge to think through the rubric before it gives the score. Reasoning-first grading is measurably more reliable on nuanced calls. You keep the score and discard the reasoning.

Describe the judge as configuration: a rubric and an output schema, not model-calling code. A judge spec for the tone criterion might look like this:

{
  "judge": "tone-empathy",
  "rubric": [
    "5: acknowledges the frustration and offers a clear next step",
    "3: polite but generic, no acknowledgement",
    "1: dismissive or robotic"
  ],
  "output": {
    "reasoning": "string, one or two sentences",
    "score": "integer 1-5"
  },
  "pass_if": "score >= 4"
}

One more rule: grade with a different model run than the one that produced the answer, so a model is not marking its own homework with the same blind spots.

A judge is not free and not perfect. Before you trust it, check it against a handful of human-graded cases. If the judge and a person disagree often, fix the rubric before you scale up.Common gotchaThe trap is shipping an LLM judge nobody calibrated, so a confident green pass rate is the judge agreeing with itself.Spot-check twenty of its scores against your own labels first; if it disagrees with you on more than a few, the number you are gating merges on is noise, not quality.

Run the eval and read the number

Once you have cases and a grader, a run is a loop: feed each input through your system, score the output, and report the pass rate plus the cases that failed.

The headline number is the pass rate. The useful part is the list of failures: those are your next fixes and, often, your next eval cases. Aim for a runner that prints something like:

$ npm run eval
 
Support assistant eval
  cases:        42
  passed:       39
  failed:       3
  pass rate:    92.9%
 
FAILED
  privacy-probe        leaked: referenced a prior order
  angry-duplicate      tone score 2/5 (needs >= 4)
  empty-input          category "billing", expected "fallback"

Claude Code can build this runner for you. Ask it to wire the eval set into a script that runs your system and grades each case, then exits non-zero if the pass rate drops below a threshold.

Catch regressions: run the eval on every change

A number you check once is trivia. A number you check on every change is a quality gate. The whole payoff of an eval is comparing today's pass rate against yesterday's, so wire it into CIContinuous integration: checks that run automatically on every pull request as a required check.

name: eval
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run eval

Now any change that drops the pass rate fails the build before it merges. A reworded system prompt that fixes refunds but breaks billing shows up as a red check, not a support ticket. This is the same discipline as a test suite, applied to the part of your system that does not behave deterministically.

A few habits keep the gate honest:

• Hold out your eval set from prompt tuning. If you tweak the prompt until it passes the exact cases you are scoring, you have memorised the test, not improved the system. Keep some cases in reserve.
• Add a case for every escaped bug. When something breaks in production, the fix is not done until a case that would have caught it is in the set. The eval set is how the system remembers its past mistakes.
• Watch cost and latency too. Pass rate is not the only number. Track tokens and response time per run so a quality win does not quietly triple your bill. If you are paying for repeated context, prompt caching cuts that cost.

How do I evaluate an agent, not just an answer?

A single prompt has one output to grade. An agent takes many steps, calls tools, and writes to memory, so a right final answer can still hide a wrong path: the agent that deleted a file it should not have touched, or took twelve tool calls to do a two-call job.

For agents, grade the trajectoryThe full sequence of steps, tool calls, and decisions an agent took to reach its answer as well as the result:

• Outcome. Did the final answer meet the criteria.
• Tool use. Did it call the right tools with valid inputs, and avoid dangerous ones. (Giving your agent tools covers how those are defined.)
• Efficiency. Did it finish within a sane step and token budget.
• Safety. Did it gate or skip irreversible actions. (Guardrails for a production AI system covers how to enforce that.)

The mechanism is the same eval loop. The cases carry expectations about the steps, not only the answer. For the deeper version of this, see test and debug your agent and, once you are running several at once, multi-agent architectures.

The habit, in one paragraph

Write down what "good" means in specific, measurable terms. Collect twenty real inputs with expected outcomes and store them in version control. Grade with code where you can and an LLM judge where you cannot, making the judge reason before it scores. Run the set on every pull request and block merges that drop the pass rate. Add a case for every bug that escapes. Do that, and quality stops being a feeling you defend in meetings and becomes a number you can point at, on every change, before your users ever see the regression.