Evals: Testing Non-Deterministic Systems

Your 2.11 changelog agent works. You ran it three times against the practice repo, read the markdown, and it looked right, so you wired it into the release pipeline. Six weeks later a teammate ships a commit that removes an API endpoint, and the agent files it under fix with a cheerful one-liner and an empty breaking array. The release notes go out. A customer's integration breaks on upgrade because the one thing that had to be flagged was the one thing the agent got wrong, on a run nobody watched. Nothing crashed. The typecheck was green the whole time. "It looked right three times" was never evidence that it was right, and now you know the difference the expensive way.

Why unit tests stop at the model boundary

Everything you have tested until now was deterministic: given the same input, the function returns the same output, so expect(f(x)).toBe(y) is a complete statement about correctness. Model output is not that. The changelog agent runs at a temperature above zero, reasons over a loop whose exact path varies, and can return two differently-worded changelogs for the same repo on two runs. There is no single golden string to assert against, because there is no single correct output, only a space of acceptable ones and a space of unacceptable ones. toBe cannot express "correctly classified the breaking change, in whatever words".

So you do not assert one output. You define success criteria, assemble a set of representative inputs with known-correct properties (the golden set), run the system against all of them, score each output against its criteria, and require the aggregate to clear a threshold. That is an eval. Anthropic's own guidance frames it as a cycle: define specific, measurable success criteria, build evals that mirror your real task distribution including edge cases, and prefer volume with automated grading over a handful of hand-graded examples. The unit test asked "is this output equal to that one". The eval asks "across inputs that look like reality, does this behavior hold often enough to trust unattended".

This is the graduation skill of the whole part. An agent you cannot eval is an agent you cannot safely give autonomy, because you have no evidence about how it behaves on the runs you are not watching. In Track B, this is literally how a background worker earns the right to run without a human on it: not because it looked right, but because it clears a golden set every time its code changes.

Three ways to grade, cheapest first

Anthropic's grading hierarchy, in the order you should reach for them:

• Code-based grading. Fastest, most reliable, infinitely scalable, and the right default. Exact match, string contains, or (for a structured agent) assertions over the fields of the output object. It cannot judge nuance, but a surprising amount of "is this correct" reduces to checkable facts: did it report the right commit range, did the removed endpoint land in breaking, is every real commit covered.
• Human grading. Most flexible, highest quality, and too slow and expensive to be your loop. Use it to build and audit the golden set, not to run it.
• LLM-based grading (LLM-as-judge). Fast, flexible, scales, and suited to the judgements code cannot make ("is this summary readable and accurate"). Powerful and dangerous in equal measure, covered below. Test its reliability before you trust it, then scale.

The engineering move is to push as much of the score as possible down into code-based grading, and reserve the judge for the residue that genuinely needs semantic judgement. Most of the changelog agent's correctness is code-checkable, which is exactly why it makes a good first eval.

Build the harness for the changelog agent

The 2.11 agent exports one function with a typed contract:

export async function generateChangelog(repoPath: string): Promise<Changelog>;
// Changelog = { range, entries: [{ kind, summary, commits }], breaking, markdown }

That typed output is what makes it evaluable in code: you are not regexing prose, you are asserting over fields. The harness is three files next to it.

The golden set. Ten cases, each a small prepared git repo (a fixture) plus the properties any correct changelog of it must have. The type carries the criteria:

import type { Changelog } from "./changelog-agent";

export type Kind = Changelog["entries"][number]["kind"];

export interface GoldenCase {
  name: string;
  fixture: string; // path to a prepared git repo for this case
  range: string; // the exact range the agent must report covering
  mustCoverCommits: string[]; // short shas that must appear in some entry
  kinds: Record<string, Kind>; // commit sha => the kind it must be filed under
  breaking: string[]; // substrings that must appear in some breaking[] item
  forbidBreaking?: boolean; // when true, breaking[] must be empty
}

export interface Check {
  name: string;
  pass: boolean;
  detail: string;
}
export interface CaseScore {
  name: string;
  checks: Check[];
  passed: number;
  total: number;
  ok: boolean;
}

export function scoreCase(output: Changelog, golden: GoldenCase): CaseScore {
  const checks: Check[] = [];
  const add = (name: string, pass: boolean, detail = "") =>
    checks.push({ name, pass, detail });

  add("entries non-empty", output.entries.length > 0);
  add("markdown non-empty", output.markdown.trim().length > 0);
  add("range matches", output.range === golden.range, `got ${output.range}`);

  const commitToKind = new Map<string, Kind>();
  for (const e of output.entries)
    for (const c of e.commits) commitToKind.set(c, e.kind);

  for (const c of golden.mustCoverCommits)
    add(`covers ${c}`, commitToKind.has(c));

  for (const [c, kind] of Object.entries(golden.kinds))
    add(`${c} => ${kind}`, commitToKind.get(c) === kind, `got ${commitToKind.get(c) ?? "missing"}`);

  if (golden.forbidBreaking) {
    add("breaking empty", output.breaking.length === 0);
  } else {
    for (const b of golden.breaking)
      add(
        `breaking notes "${b}"`,
        output.breaking.some((x) => x.toLowerCase().includes(b.toLowerCase())),
      );
  }

  const passed = checks.filter((c) => c.pass).length;
  return { name: golden.name, checks, passed, total: checks.length, ok: passed === checks.length };
}

Every check is a fact about the structured output, which is why the schema you fought for in 2.11 pays off here: scoring a typed object is code, scoring free prose would be another model call. This scorer was executed against sample outputs during authoring: a fully-correct changelog scores 10 of 10, and a changelog with a wrong range, a misfiled fix, a missing commit, and a dropped breaking change scores exactly 5 of 10 with each miss named (five failed checks from four defects: the missing commit fails both the coverage check and the kind check). That is the point of the detail field: a failing eval tells you which property broke, not just that something did.

The golden cases themselves. The set only means something if it mirrors the real distribution and includes the edge cases where the agent actually fails. Ten cases spanning what a changelog agent meets in the wild:

Cases 2 and 9 are the ones that would have caught the cold open. A golden set without them is a golden set that agrees with your agent's blind spots.

The runner and the CI gate. One script runs every case, scores it, and fails the process when the aggregate falls below threshold:

import { generateChangelog } from "./changelog-agent";
import { scoreCase, type CaseScore } from "./score";
import { GOLDEN } from "./golden";

const PASS_THRESHOLD = 0.9; // 90% of all checks across the suite must pass

async function main() {
  const scores: CaseScore[] = [];
  for (const golden of GOLDEN) {
    const output = await generateChangelog(golden.fixture);
    const score = scoreCase(output, golden);
    scores.push(score);
    console.log(`${score.ok ? "PASS" : "FAIL"} ${score.name} ${score.passed}/${score.total}`);
    for (const c of score.checks) if (!c.pass) console.log(`   miss: ${c.name} ${c.detail}`);
  }

  const passed = scores.reduce((n, s) => n + s.passed, 0);
  const total = scores.reduce((n, s) => n + s.total, 0);
  const rate = passed / total;
  console.log(`\nsuite: ${passed}/${total} checks (${(rate * 100).toFixed(1)}%)`);
  if (rate < PASS_THRESHOLD) {
    console.error(`FAIL: below threshold ${PASS_THRESHOLD * 100}%`);
    process.exitCode = 1;
  }
}

main();

The threshold is a design decision, not a default. A perfect 100% bar makes the suite flaky, because the model varies run to run and one stylistic miss on one case reddens CI for no real regression. Too low a bar passes real breakage. Set it from a baseline: run the suite several times on the known-good agent, see where an honest pass lands (say 94 to 98 percent), and put the threshold below your worst good run and above where real regressions drag it. For the properties that must never fail (case 2's breaking-change detection), promote them to their own hard assertion so a single miss fails the suite regardless of the aggregate.

Wire it into CI as a merge gate, the same way typecheck and lint are gates:

name: evals
on: [pull_request]
jobs:
  changelog-evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: "22" }
      - run: npm ci
      - run: npx tsx run-evals.ts
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Now a change to the agent's prompt, its schema, or its model cannot merge unless the golden set still passes. This is the regression eval: the golden set is the contract, and CI enforces it on every change, so "I tweaked the prompt and it still works" stops being a hope and becomes a checked fact.

The judge, for what code cannot score

Code checks that the breaking change was flagged. It cannot check whether the markdown reads well or whether a summary is accurate to its commits. That residue is where LLM-as-judge earns its place: a separate, cheap model call that scores the prose against a rubric.

// A plain API call (not an agent): the judge only reads and scores.
// Rubric is explicit and ordinal so the score is comparable across runs.
const JUDGE_RUBRIC = `
Score the changelog markdown 1-5 on accuracy and readability.
5: every entry is accurate to its commits and a reader instantly understands the release.
3: mostly accurate, some vague or padded entries.
1: inaccurate or unreadable.
First reason in one sentence, then output only: SCORE: <n>.
`;

Anthropic's guidance for a usable judge is specific: give it a detailed rubric with concrete pass conditions, make the output empirical (a fixed scale or a correct/incorrect label, never a vibe), and have it reason briefly before scoring, then discard the reasoning. You keep the number, not the essay.

Where it breaks

The grader that can never fail. The most dangerous eval is one that passes everything, because it retires your suspicion while proving nothing. A golden set with no case 2 and no case 9, a threshold set at 40 percent, a judge rubric so loose every output scores a 4: each is a gate that a broken agent sails through. This is the reward-hacking lesson from 1.5 turned on the grader itself. The test of a golden set is whether it can fail: deliberately break the agent (force every kind to feature, as the 2.11 lab did) and confirm the suite goes red. A suite that stays green on a sabotaged agent is measuring nothing.

LLM-as-judge failure modes, which is why it is never unaudited. The judge is a model, so it inherits model failures, and the position from this course is blunt: LLM-as-judge is useful and must never be trusted unaudited. Known failure modes to design against:

• Self-preference and position bias. A judge tends to favor outputs from its own model family, and in pairwise comparisons favors whichever answer is shown first. If you judge, randomize order and do not let the judged model grade its own homework unsupervised.
• Sycophancy and length bias. Judges reward longer, more confident-sounding output even when it is padded or wrong, the same tendency 1.5 named in the agent itself.
• Rubric drift. A vague rubric lets the judge's interpretation wander run to run, so the "same" output scores 4 then 2. Anthropic's fix is the empirical, specific rubric above; the fuzzier the rubric, the less the score means.

The audit is not optional and it is cheap: periodically hand-grade a sample of the cases the judge scored, and check that human and judge agree. When they diverge, the judge is wrong until proven otherwise, and the rubric gets tightened. A judge you have never audited is a random number generator you have chosen to believe.

Overfitting the golden set. Ten cases you tuned the prompt against can turn into ten cases the prompt has memorized, passing the suite while the real distribution still fails. Rotate in new cases from real runs, especially every failure you find in production, so the golden set grows toward reality instead of toward your agent's current behavior.

Knowledge check

Sources

• Define success criteria and build evaluations (grading hierarchy: code-based, human, LLM-based; eval design principles: task-specific, automate, volume over quality; LLM-judge tips: detailed rubrics, empirical/specific output, reason-then-discard): https://docs.claude.com/en/docs/test-and-evaluate/develop-tests (fetched July 2026)
• Using the Evaluation Tool in Console (test sets, prompt iteration against graded cases): https://docs.claude.com/en/docs/test-and-evaluate/eval-tool (fetched July 2026)
• Agent SDK structured outputs (the typed Changelog contract that makes code-based scoring possible; consistent with lesson 2.11's agent): https://code.claude.com/docs/en/agent-sdk/structured-outputs (fetched July 2026)
• The score.ts logic was executed against sample correct and broken changelog outputs during authoring (correct scores 10/10; a wrong-range, misclassified, missing-commit, dropped-breaking output scores 5/10 with each miss named), verified July 2026.