Advanced Evals

Two weeks after lesson 3.7's harness went in, you tweak the changelog agent's classification prompt to fix a misfiled refactor. You run npx tsx run-evals.ts before and after: 8 of 10 golden cases clean before, 9 of 10 after. You write "improved classification accuracy" in the PR description. Here is the uncomfortable arithmetic: on a ten case suite, one run each, 8/10 versus 9/10 is statistically indistinguishable from no change. Fisher's exact test on those two results gives a p value of 1.0. Pure noise, dressed as progress. This appendix is about what your harness can honestly claim, and how to grow it until it can claim more.

What ten cases can and cannot tell you

The 3.7 suite was the right first artifact: a golden set, a code scorer, a threshold, a CI gate. What it is not is a precision instrument, and the failure mode of teams that build their first eval is treating it like one.

Your observed pass rate is a sample estimate of a true underlying rate you cannot see: the rate the agent would achieve across every input shaped like your distribution. Anthropic's statistics paper on evals frames it exactly this way, questions drawn from an unseen question universe, with the observed average scattering around the true one. How much scatter? The standard error of a proportion is the square root of p times (1 minus p) over n. At 8/10 that is the square root of 0.8 times 0.2 over 10, which is 0.126. Multiply by 1.96 for a 95 percent interval and you get plus or minus 25 percentage points. The normal approximation is shaky at n of 10, so use the Wilson interval instead, which gives:

• 8/10 passing: the true rate sits somewhere between roughly 49 and 94 percent
• 9/10 passing: between roughly 60 and 98 percent

Those intervals nearly cover each other. Concretely: an agent whose true pass rate is a mediocre 80 percent will score 9/10 or a perfect 10/10 on about 38 percent of runs, just by drawing lucky. And to reliably detect a real improvement from 85 to 95 percent (5 percent significance, 80 percent power), you need on the order of 140 cases per comparison. Nobody hand-builds 140 git fixtures, and you should not.

So take the position the arithmetic forces: a small golden set is a tripwire, not a meter. It reliably catches collapses, an agent whose true rate fell from 90 to 50 percent will almost never sneak past ten cases, and that is precisely what a merge gate needs to catch. What it cannot do is rank two prompts that differ by a few points, and any dashboard that trends a ten case suite to one decimal place is theater.

Two upgrades make a small set honest without 140 fixtures. First, trials: run each case k times instead of once. Per-run behavior varies (lesson 3.7 already made you set the threshold from multiple baseline runs), so the per-case pass rate across trials is the real signal. The agent evals guidance from Anthropic's engineering team formalizes this as pass@k, the chance at least one of k trials succeeds, versus pass^k, the chance all k succeed. Pass^k is the bar for anything unattended: a case your agent passes 90 percent of the time has only a 59 percent chance of passing five straight trials (0.9 to the fifth power). Flakiness you never saw at one trial each becomes visible at five. Second, paired reading: when you compare prompt A to prompt B, do not compare aggregates. Compare the same cases under both, list which flipped, and read those transcripts. At small n, one honestly read transcript outranks any percentage.

Eval-driven development: the failure becomes a case first

Lesson 3.7 told you to rotate production failures into the golden set. Eval-driven development sharpens that from housekeeping into the first step of every fix: before you touch the prompt, reproduce the failure as a golden case and watch it go red.

The reason is the same one that justified test-driven development, plus one that is specific to agents. Prompt changes are global: an instruction added to fix commit classification also shifts how the agent summarizes, ranges, and formats. Without a red case that turns green, you cannot distinguish "fixed the failure" from "moved it somewhere the suite does not look." And the red-first step doubles as a grader audit: if you cannot make the observed failure turn a case red, your scorer cannot see that class of failure, and the fix belongs in score.ts before anything else. Anthropic's agent evals guide puts the matching maintenance rule bluntly: read the transcripts, because failures should seem fair, and a score you cannot explain is a score you cannot trust.

In practice: trials before verdicts

The upgrade to lesson 3.7's runner is small. Same generateChangelog, same scoreCase, same GOLDEN, same threshold; the loop gains a TRIALS knob and per-case pass rates, and writes a scores file you can trend:

import { generateChangelog } from "./changelog-agent";
import { scoreCase } from "./score";
import { GOLDEN } from "./golden";
import { writeFileSync } from "node:fs";

const TRIALS = Number(process.env.TRIALS ?? "1");
const PASS_THRESHOLD = 0.9;

async function main() {
  const perCase: { name: string; passed: number; trials: number }[] = [];
  let checksPassed = 0;
  let checksTotal = 0;

  for (const golden of GOLDEN) {
    let passed = 0;
    const misses = new Map<string, number>();
    for (let t = 0; t < TRIALS; t++) {
      const output = await generateChangelog(golden.fixture);
      const score = scoreCase(output, golden);
      checksPassed += score.passed;
      checksTotal += score.total;
      if (score.ok) passed++;
      else
        for (const c of score.checks)
          if (!c.pass) misses.set(c.name, (misses.get(c.name) ?? 0) + 1);
    }
    perCase.push({ name: golden.name, passed, trials: TRIALS });
    console.log(`${golden.name}: ${passed}/${TRIALS} trials clean`);
    for (const [name, count] of misses) console.log(`   miss x${count}: ${name}`);
  }

  const rate = checksPassed / checksTotal;
  console.log(`\nsuite: ${checksPassed}/${checksTotal} checks (${(rate * 100).toFixed(1)}%) over ${TRIALS} trial(s)`);
  writeFileSync(
    "eval-scores.json",
    JSON.stringify({ date: new Date().toISOString(), trials: TRIALS, rate, perCase }, null, 2),
  );
  if (rate < PASS_THRESHOLD) {
    console.error(`FAIL: below threshold ${PASS_THRESHOLD * 100}%`);
    process.exitCode = 1;
  }
}

main();

Ten cases at five trials is fifty agent runs, which is real money and real minutes. That cost split is what the CI section below is for: one trial on every PR, full trials on a schedule.

Calibrating the judge, and deciding if it deserves the effort

Lesson 3.7 gave you judge.ts for the residue code cannot score, and its audit rule was directional: hand-grade a sample, check that human and judge agree. Calibration makes the audit a number, because "we mostly agree" hides a trap.

Say you hand-grade 20 changelog outputs as pass or fail on readability, blind, before looking at any judge scores. The judge (mapping its 1 to 5 scale: 4 and up is pass) agrees with you on 17 of 20, 85 percent. Sounds strong. But if both you and the judge pass about 80 percent of outputs, two graders flipping weighted coins with those base rates would agree 68 percent of the time by pure chance (0.8 times 0.8 plus 0.2 times 0.2). Cohen's kappa corrects for exactly that: observed agreement minus chance agreement, over one minus chance agreement. Here that is 0.17 over 0.32, a kappa of 0.53. Moderate, not strong. The judge is adding real signal over a coin, but not enough that you would let it fail a merge on its own.

import { readFileSync } from "node:fs";

type Verdict = "pass" | "fail";
interface Labeled {
  name: string;
  human: Verdict;
  judge: Verdict;
}

const rows: Labeled[] = JSON.parse(readFileSync("labels.json", "utf8"));
const n = rows.length;

const agree = rows.filter((r) => r.human === r.judge).length / n;
const humanPass = rows.filter((r) => r.human === "pass").length / n;
const judgePass = rows.filter((r) => r.judge === "pass").length / n;

const chance = humanPass * judgePass + (1 - humanPass) * (1 - judgePass);
const kappa = (agree - chance) / (1 - chance);

console.log(`n=${n} raw agreement ${(agree * 100).toFixed(0)}%`);
console.log(`base rates: human pass ${(humanPass * 100).toFixed(0)}%, judge pass ${(judgePass * 100).toFixed(0)}%`);
console.log(`chance agreement ${(chance * 100).toFixed(0)}%, kappa ${kappa.toFixed(2)}`);

for (const r of rows.filter((x) => x.human !== x.judge))
  console.log(`disagree: ${r.name} human=${r.human} judge=${r.judge}`);

My working bar: kappa above roughly 0.6, the judge may gate; between about 0.4 and 0.6, it advises (a low score triggers a human read, never a red build); below 0.4, rewrite the rubric or drop the judge. And calibration is not a one-time ceremony. Rerun it whenever the rubric changes, the judge model changes, or the agent's output style drifts, because every one of those silently moves what the score means. Anthropic's agent evals guidance lists calibration against human graders as a standing requirement of model-based grading, not a nice-to-have, and lists human labels as the thing model graders are calibrated from.

Which raises the sharper question: is the judge worth this at all? Twenty blind labels plus periodic recalibration is hours of your attention. The position from lesson 3.7 still wins here: any property you can restate as a code assertion should be one, because code costs nothing to rerun and never drifts. A judge earns its calibration overhead only when the property is genuinely semantic (readability, faithfulness of a summary to its commits) and the volume is high enough that you cannot just read the outputs yourself. For the changelog agent at ten cases, honestly, reading is competitive with judging. At two hundred nightly runs, it is not, and the calibrated judge pays for itself.

Grading behavior, not just output

Everything so far scores the Changelog object: the outcome. Multi-turn agents also have a trajectory, the sequence of tool calls and intermediate decisions that produced the outcome, and some properties live only there. Anthropic's grader taxonomy for agents makes the split explicit: graders evaluate some portion of either the transcript or the outcome, and its code-based grader list includes tool call verification (which tools ran, with what parameters) and transcript analysis (turns taken, tokens spent) alongside output checks.

Case 4 in your golden set is the tell. It exists to verify the agent reads changed files when commit messages are vague instead of guessing. But an outcome check cannot distinguish reading from lucky guessing, and at one trial a guesser passes it often. Trials help (guessers flunk pass^k). A trajectory check settles it: assert that the transcript contains a file-read tool call before any classification was produced. Same with case 10's turn budget, which was always a trajectory property wearing an outcome costume.

Two rules keep trajectory grading from eating you. First, grade the outcome wherever the outcome is checkable, and reach into the trajectory only when the path is the property, budgets, forbidden tools, required evidence-gathering. Second, assert the minimum, never the exact path. An eval that pins the precise tool sequence fails every valid variation, which Anthropic flags as the core weakness of code graders, and frontier models make it worse by finding better paths than yours: Opus 4.5 famously "failed" a tau2-bench flight-booking task by finding a legitimate policy loophole that solved the customer's actual problem. An eval that punishes better solutions trains you to ship worse agents.

CI at scale: the tripwire and the trend

Lesson 3.7 gave you evals.yml as a PR merge gate, and lesson 2.9 gave you the cost discipline for anything scheduled. At scale the pattern splits into two jobs with two different questions:

• The PR gate asks: did this change break anything? One trial per case, hard assertions on the must-never-fail properties (case 2's breaking-change detection), threshold on the rest. Fast, cheap, and per the arithmetic above, honest: a collapse trips it, and a collapse is what merges cause.
• The nightly asks: what is the trend? Full trials (five per case), writing eval-scores.json somewhere durable, per-case pass rates over days. This is where a real 90-to-80 drift becomes visible long before any single PR run would show it, and where a model version bump gets re-baselined before you blame your own prompt.

name: evals-nightly
on:
  schedule:
    - cron: "0 6 * * *"
  workflow_dispatch:
jobs:
  changelog-evals:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: "22" }
      - run: npm ci
      - run: npx tsx run-evals.ts
        env:
          TRIALS: "5"
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Publish eval-scores.json wherever your team keeps history (a workflow artifact, a committed history file, a dashboard that shows intervals rather than false precision). Watch cost the 2.9 way: fifty runs a night times total_cost_usd is a number someone should see weekly. When the harness outgrows a hand-rolled runner, because you want prompt-by-model matrices or a second agent shares the fixtures, promptfoo is the established open source option (actively maintained as of July 2026, declarative configs, CI integration, and it evaluates agents, not just prompts). Reach for it for the matrix features, not because a 150-line runner you fully understand stopped working. It did not.

Where it breaks

Dashboard theater. The most common advanced-eval failure is not bad statistics, it is confident presentation of no-signal numbers.

The calibrated-once judge. A judge calibrated in March against the March rubric and the March model is uncalibrated by July; you just have no number telling you so. Kappa decays silently every time anything under it moves. Recalibrate on every rubric edit and every judge-model change, and spot-check quarterly regardless.

Eval saturation. A suite that reads 100 percent for six straight weeks feels like victory and provides zero improvement signal; it can only tell you about regressions. Anthropic calls this eval saturation, and its guidance is the one you already practice: keep feeding real production failures in as new red cases, so the frontier of the suite tracks the frontier of what your agent still gets wrong. A golden set that never gains a case is aging into flattery.

Trajectory over-specification. Covered above, and worth its own tombstone: every trajectory assertion is a bet that your path is the only valid path. Lose that bet often enough and your team starts ignoring red suites, which is the terminal disease of any gate.

Knowledge check

Sources

• Demystifying evals for AI agents (Anthropic Engineering): https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents (fetched July 2026)
• A statistical approach to model evaluations (Anthropic): https://www.anthropic.com/research/statistical-approach-to-model-evals (fetched July 2026)
• Define success criteria and build evaluations (Anthropic): https://docs.claude.com/en/docs/test-and-evaluate/develop-tests (fetched July 2026)
• promptfoo (repo): https://github.com/promptfoo/promptfoo (fetched July 2026)