Measurement, Iteration, and Handoff

Eight weeks in, Meridian's founder asks the only question that keeps you hired: "Is this actually working, and can it run if you disappear?" A gut answer loses the account. You answer with a scorecard, a graduation decision backed by evals, and a handoff package that lets Meridian operate the system without you, because a consultant who is a single point of failure is a liability the client is right to worry about. This lesson turns the running system into proof of value, a plan to improve it, and a service you can price and sell. The Agent OS is done when it survives you.

The per-process scorecard

You cannot manage what you do not measure, and you cannot sell what you cannot prove. Every process gets a scorecard, built from the run-log and audit log you already made first-class in 5.6 and 5.7, so the data collection was designed in, not reconstructed after the fact. Four numbers per process, and each maps to something the buyer cares about.

• Runs. How many times it executed, by trigger and by status. Volume is the denominator for everything else.
• Draft acceptance rate. Of the drafts a human reviewed, what fraction were sent as-is or with only minor edits. This is the single most important number: it is the evidence that moves a process up the autonomy dial, and it is the operational read on quality.
• Minutes saved. Runs times minutes returned per run, the 5.1 ROI formula measured rather than estimated. This is the number on the renewal conversation.
• Incidents. Wrong output that reached a client, or a failure that caused harm. Distinct from a caught-and-contained failure, which is the system working. Incidents are the number that must stay at zero for autonomy to advance.

A real eight-week scorecard for Meridian's report drafter, drawn from the logs:

# Scorecard: Report Drafter (weeks of 2026-05-04 through 2026-06-22)

Runs:            48  (6 clients x 8 weeks; 46 success, 2 failed)
Draft acceptance: 83%  (40 of 48 sent as-drafted or minor edits)
Minutes saved:   ~56 hours total (~70 min/report x 48), ~7 hrs/week
Incidents:       0 client-facing. 2 failed runs (missing Ads CSV)
               escalated correctly, no wrong report sent.
Autonomy:        Draft -> recommend Approve (see graduation below)

Top rejection reasons (the 8 edited/rejected drafts):
- 4x: narrative buried a flagged metric the AM wanted led with
- 2x: tone too hedged on a good week (voice.md gap)
- 2x: partial-data week phrased awkwardly

That card is a sales asset and an engineering to-do list at once. Seven hours a week returned, zero incidents, and a specific, short list of what the rejections were actually about.

Graduate autonomy on evidence, not comfort

Moving a process up the dial (5.1) is a decision with a rule, and the rule is the 3.7 eval pattern applied to operations. A process graduates from Draft to Approve, or from Approve to Autonomous-with-audit, only when three things are all true, consistently, over a real window:

• The evals pass. A golden set for the worker, built exactly as 3.7 built one for the changelog agent: representative inputs (a clean week, a missing-data week, the tracking-error week, a client with quirks) with known-correct properties, scored in code against the typed output contract, gating on a threshold set from baseline runs. The report drafter's evals check that deltas are correct, the over-50% move is flagged not narrated, missing data is listed not invented, and no cross-client data appears. An agent that cannot clear its golden set does not graduate, full stop, the same non-negotiable as 3.7's CI gate.
• The acceptance rate clears the bar. A high, stable draft-acceptance rate over enough runs means editing has become rubber-stamping, which is the operational signal that approve-with-one-click is safe. The report drafter's 83% over 48 runs is the kind of evidence that earns Approve; a jumpy rate does not.
• Incidents are zero. One client-facing incident resets the clock. Autonomy is earned back, not assumed.

The report drafter, at 83% acceptance, a passing golden set, and zero incidents, graduates to Approve. The lead triage worker, external-facing and higher blast radius, stays on Draft longer even at a similar acceptance rate, because the cost of its rare miss is a mis-sent prospect email, and the dial respects blast radius, not just the average. Graduation is evidence meeting a threshold, never the client's comfort or your optimism, which is the same discipline that kept a worker off autonomy in 5.1 and kept code from shipping unverified in Part 3.

Iterate the spec from rejection reasons

The rejection reasons on the scorecard are the highest-value data the system produces, because they are real humans telling you exactly where the worker is wrong, for free, every week. You do not guess at improvements; you read the rejections and fix the spec. The report drafter's top rejection ("narrative buried a flagged metric the AM wanted led with") is not a model problem, it is a spec gap: the process spec never said flagged metrics lead the narrative. So you edit sops/weekly-report.md to say exactly that, add a golden case that checks it, and confirm the eval catches the old behavior. The "tone too hedged on a good week" rejections point at a voice.md gap, fixed the same way. This is the eval-driven development loop from 3.7 running on operations: rejection reasons become golden cases, the spec tightens, the acceptance rate climbs, and each improvement is a reviewable commit to the knowledge repo, not a silent prompt tweak. A spec that stops changing is either perfect or unmonitored, and it is never perfect.

The handoff package

The engagement is not the build, it is a system Meridian can operate. That means a handoff package, five documents, each of which makes the client less dependent on you existing.

• System map. The five layers for Meridian on one page: what each worker does, what it reads and writes, what triggers it. The 5.1 figure instantiated with Meridian's real workers.
• Runbooks. For each worker: how to run it manually, how to read its scorecard, what its common failures look like and what to do, how to change its spec. Written so Grace can follow them without you.
• Access inventory. Every credential the system uses, its scope, where it is stored, who owns rotating it. The 5.4 matrix made operational, so an audit or an offboarding is a document lookup, not an archaeology dig.
• Scorecard. The living one, with how it is generated from the logs, so Meridian can read value monthly without you.
• Escalation path. Who gets paged when a worker fails, through which surface, and what the safe-state guarantees are (5.7). The 2am-break plan, written down.

The test of the package is blunt: could Meridian keep the system running for a quarter if you vanished? If not, you have sold a dependency, not a system, and the client will feel it the first time something breaks and you are on vacation.

Consultant framing: pricing and the sales conversation

An Agent OS is not a one-time build you hand over and leave, it is an operated service, and pricing it as a build undercharges for the part that has the ongoing value: running it, watching the logs, iterating specs, owning freshness. Price it in two parts.

• Discovery and build, one-time. Process mapping, the knowledge repo, the workers, the wiring. Scoped to the number of processes. This covers your build cost and is the smaller number.
• Operated service, monthly. You run the system: monitor the run-log, review the audit log, keep the golden sets current, iterate specs from rejection reasons, own knowledge freshness, and hold the escalation pager. This is the recurring number, and it is justified by the recurring value.

The pricing logic anchors to value, not to your hours. Meridian's report drafter alone returns about seven hours a week; across the three processes the returned time and the faster lead response are the number the monthly fee is measured against, and a fee that is a fraction of the value returned is an easy yes. You do not price by how long the build took you, because the client does not buy your hours, they buy their returned time and their avoided errors.

The sales conversation follows from everything in this part, and it is not a pitch, it is a demonstration. You lead with the dial, because the nervous buyer's fear is the honest engineer's discipline: "Nothing sends without your approval until it has earned it, and here is the evidence that moves it." You show the scorecard: real runs, real acceptance rate, real hours returned, zero incidents. You show the caught injection, because "here is the attack we already stopped" is worth more than any promise of safety. And you offer a paid pilot on one process, priced small, because a scorecard on one real process in three weeks closes better than a deck ever will. You are not selling AI. You are selling returned hours, avoided errors, and a system that proves both, operated by someone who stays accountable for it.

Knowledge check

Sources

• Define success criteria and build evaluations (golden-set evals as the graduation gate; measurable success criteria; the 3.7 pattern applied to graduating a worker's autonomy on evidence): https://docs.claude.com/en/docs/test-and-evaluate/develop-tests (fetched July 2026)
• Agent SDK structured outputs (the typed output contracts that make the scorecard's acceptance and incident measures scorable in code, consistent with 2.11 and 3.7): https://code.claude.com/docs/en/agent-sdk/structured-outputs (fetched July 2026)