3.10 · Calibrating Rigor: Properly AND Quickly

3.10 · 20 min

Calibrating Rigor: Properly AND Quickly

Two engineers, same afternoon. The first wraps a forty-line dispatch, a goal tree, and a verification registry around a script that renames a config key, and spends an hour on ceremony for a change git could undo in one command. The second skips all of it for the new billing webhook handler, vibes it into place, watches it pass a happy-path click-through, and ships. One over-armored a paper cut; the other under-armored a production money path. Both mistakes are the same mistake: rigor set by habit instead of by the task. This lesson makes rigor a dial you set deliberately, and it names the single most expensive miscalibration in agentic work, the one that quietly builds every house of cards: treating a prototype as something you can promote.

Rigor is a dial, not a personality

Everything in Part 3 (specs, gates, review, evals, the harness) has a cost in time and attention. The mistake is treating that cost as either always worth paying or never worth paying. It is worth paying in proportion to what a mistake costs, and that is a computation, not a temperament. Three inputs set the dial:

  • Stakes. What does a wrong result cost? A throwaway analysis script that misreads gets rerun. A billing handler that miscounts charges customers wrong and burns trust. Higher stakes, more rigor.
  • Reversibility. How hard is it to undo? A local file edit reverts in a keystroke. A migration that has run against production data, an email blasted to your list, a payment captured: these do not revert, and irreversibility demands rigor up front because there is no cheap fix after.
  • Verifiability. How easily can you prove it is correct? If a glance confirms it, ceremony adds little. If correctness lives in behavior you cannot eyeball (concurrency, money math, multi-tenant isolation, model output), you need the machinery that makes correctness observable, because "looks right" is worth nothing exactly where it is hardest to check.

Set the dial from the highest of the three. A change that is low-stakes and reversible but genuinely hard to verify still needs real checks. The discipline is to run the computation before you start, out loud, in one sentence: "this is reversible and low-stakes, tier 1, typecheck only" or "this touches money and is irreversible, tier 3, full harness plus a security review." Naming the tier before you start is the whole skill, because the expensive errors happen when nobody decided and the default just happened.

The three tiers

The dial has three useful detents. Each is a bundle of the Part 3 machinery, sized to the task.

  • Tier 1, throwaway (prototype mode). Goal: learn something or produce a disposable artifact, fast. Skip planning, skip the dispatch ceremony, skip tests. Keep exactly one gate: the typecheck, because a prototype that does not compile teaches you nothing and costs nothing to keep green. Vague prompts are correct here; the docs endorse them precisely because you can afford to course-correct when the work is exploratory and cheap to throw away. Examples: a scratch script to check an API's shape, a UI spike to feel a layout, a one-off data pull.
  • Tier 2, standard. Goal: real work that will live in the codebase. The default working mode from 3.1 and 3.2: a dispatch with all five sections, the machine gates (typecheck, lint, unit, integration as they apply), a diff review against scope. Most feature work is here.
  • Tier 3, critical. Goal: work where a mistake is expensive or irreversible. Everything in tier 2 plus the machinery this part built for exactly these cases: evals for any non-deterministic behavior (3.7), a security review for anything touching untrusted input or auth (3.8), and the full harness (3.9) when it spans sessions. Auth, payments, migrations, multi-tenant data access, and anything an unattended agent runs live here. Briefcase's RLS (4.5), auth (4.6), and payments (4.7) are all tier 3, which is why those lessons gate on tested attacks, not on "it worked when I clicked it."

The tiers are not a ladder you climb; they are a selection you make. Most of the waste in practice is tier-1 ceremony on tier-1 work (the renamed config key) and tier-1 casualness on tier-3 work (the billing handler). Getting the tier right is worth more than doing any single tier perfectly.

The expensive one: prototype-to-production is a rewrite

Here is the position this lesson exists to argue, and it is the one most people resist: you do not promote a prototype to production. You rewrite it, at the correct tier, using the prototype as the spec. The prototype's job was to teach you what to build; its code is not the thing you ship. Promoting it, keeping the tier-1 code and "hardening" it in place, is the single most common source of the house-of-cards systems 3.3 warned about.

Why promotion feels right and is wrong: the prototype runs. It demos. It looks ninety percent done, so finishing it looks like a small job. But the tier-1 code has no tests (so you cannot change it safely), no scoped permissions or input validation (so it is an injection surface, 3.8), no error handling on the paths you did not happen to hit, and an architecture optimized for reaching a demo fast, not for being legible to the agents who will now maintain it (3.3). "Hardening in place" means retrofitting all of that onto code that was never shaped for it, blind, because you have no tests to tell you when a change breaks something. Every retrofit is a guess, and the guesses stack.

Promote the prototype

Keep the tier-1 code, "harden" it in place: bolt on tests around code not written to be testable, add validation and error handling into paths you now have to reverse-engineer, patch the architecture toward something maintainable. You are changing untested code with no gate to catch regressions, so every fix risks a new break you will not see until production does. This is the house of cards: it stands until the one card you touch.

Rewrite from the prototype as spec

Freeze the prototype as an executable spec (it already answers "what does this do and how should it behave"), then rebuild at the correct tier: a real dispatch, a data model designed on purpose, tests written alongside the code, scoped permissions, the gates. The prototype told you exactly what to build, so the rewrite is fast and informed, and the result is code shaped for change from line one.

The cost math

The instinct against rewriting is "that is throwing away working code." Run the numbers and the instinct inverts. Take a prototype that took 4 hours to reach a working demo.

  • Promote-and-harden. Retrofitting tests onto untestable code, reverse-engineering the paths you skipped, and patching the architecture typically runs longer than the original build, because you are working blind against code not shaped for it: call it 6 to 10 hours. Then the tax that does not show up on the invoice: every future change to this module is slower and riskier because the foundation is patched rather than designed, and some fraction of those changes ship a regression the missing tests would have caught. One production incident from a data leak or a billing error can cost more than the entire project, and tier-1 code on a tier-3 path is how that incident gets in.
  • Rewrite from spec. You already know exactly what to build (the prototype is the spec) and you are working at tier 2 or 3 from the start, so the rebuild is a clean, well-specified 3 to 5 hours, and it ends with tests, a sane architecture, and a gate. Future changes are cheap and safe because the foundation was designed, not retrofitted.

The rewrite is usually cheaper in the first week and dramatically cheaper over the life of the code, and it removes the tail risk (the production incident) that dominates the expected cost on a tier-3 path. "Throwing away working code" is the wrong frame: you are throwing away four hours of code and keeping the four hours of learning, which was the whole point of the prototype. The learning is the asset; the code was scaffolding.

Where it breaks

Tier inflation as procrastination. Reaching for the full harness on genuinely trivial work is not diligence, it is avoidance dressed as rigor. If the change is reversible, low-stakes, and glance-verifiable, tier 1 is the correct, professional choice, and the forty-line dispatch for a rename is a failure of calibration in the expensive direction.

Silent tier drift upward. A tier-1 prototype that keeps getting "just one more feature" is being promoted by accretion, without anyone deciding to. The moment a throwaway starts being relied on, its tier changed and the rewrite is now due; the failure is not noticing the moment. Name it when a spike stops being a spike.

Confusing tier with size. A ten-line change to an RLS policy is tier 3; a four-hundred-line change to a scratch analysis notebook is tier 1. Rigor tracks stakes, reversibility, and verifiability, never line count. The smallest diffs on the highest-stakes paths are exactly where casual tiering ships disasters.

Lablab-3-10Same feature, two tiers

Goal: Build one small feature twice, at tier 1 and tier 2, and measure the difference in artifacts, time, and what each version can safely survive.

Prereqs: the agentic-practice repo, and the harness artifacts from 3.9 available (you will use the tier-2 machinery).

  1. Pick a small but real feature with a checkable behavior, for example: "a --since <date> filter on the repo-stats script that limits counts to files changed since that date." State the behavior in one sentence.
  2. Build it tier 1 first. No dispatch, a vague prompt ("add a --since date filter to repo-stats"), no tests, keep only the typecheck green. Time it from prompt to working demo. Note what you did NOT do: no scope, no tests, no edge cases (bad date format, future date, no matching files).
  3. Note the tier-1 gaps honestly: try a malformed --since notadate and an empty result set. Record what breaks or misbehaves, and that you have no test that would have caught any of it.
  4. Now build the SAME feature tier 2, in a fresh session, treating the tier-1 version as your spec (it told you exactly what the feature is). Write a real dispatch (five sections), write tests first or alongside including the edge cases from step 3, run the full gate, review the diff against scope. Time this too.
  5. Compare, in writing: the artifacts each produced (tier 1: a diff and a green typecheck; tier 2: a dispatch, tests, a reviewed diff, a full gate), the wall-clock time, and crucially what each version can survive: hand each to a fresh session and ask it to add a related feature. The tier-2 version's tests catch the regression the change introduces; the tier-1 version's silence does not.
  6. Write the calibration verdict: for this feature in a real product, which tier was correct and why, in terms of stakes, reversibility, and verifiability. Then state the rewrite framing explicitly: the tier-1 build was a prototype, the tier-2 build was the real thing, and the prototype served as its spec, not its foundation.

Verify

  • Two working versions of the same feature exist, one tier 1 (typecheck only) and one tier 2 (dispatch, tests, full gate), each with its build time recorded.
  • You documented at least two behaviors the tier-1 version gets wrong or unsafely (e.g. malformed date, empty result) and confirmed no tier-1 test would catch them.
  • When a fresh session extends each version, the tier-2 tests catch a regression the tier-1 version ships silently. You observed this, not assumed it.
  • A written verdict names the correct tier for this feature from stakes/reversibility/verifiability, and states the prototype-as-spec, rewrite-not-promote framing in your own words.
>Troubleshooting
  • The tier-1 version took nearly as long as tier 2: you were sneaking in rigor (tests, edge cases) the tier does not include. Let tier 1 be genuinely minimal; the point of the exercise is to feel the real gap, including the risk the speed buys.
  • You cannot find a regression for step 5: pick an extension that touches shared logic (change how a date is parsed, or how counts aggregate) so the tier-2 tests have something to catch. If nothing can regress, the feature is too trivial to teach the lesson; pick one with real behavior.
  • The rewrite felt slower than expected, not faster: check that you actually used the prototype as the spec instead of re-deriving the feature. The speedup comes from already knowing what to build; if you re-explored, you paid the discovery cost twice.

Knowledge check

Knowledge check

Q1You are about to change a row-level-security policy: a ten-line diff on a multi-tenant app. A colleague says 'it's tiny, just ship it after a quick look.' What tier is this and why, and what does the size tell you?
Q2A four-hour prototype demos well and looks nearly done. Your PM wants to 'just harden it' for production. What is the lesson's position, and the core reason?
Q3A colleague argues that rewriting a working 4-hour prototype 'throws away 4 hours of work,' so hardening in place must be cheaper. Where does the cost math actually land?
CheckpointCheckpoint 3: A full-discipline run

This is the midpoint of the course, and the integrative test of everything in Part 3. Take one real feature for your practice repo, small but not trivial, and run it through the complete discipline end to end, using the harness you installed in 3.9.

Name the tier before you start. Write the dispatch (3.1). Set and run the gate (3.2). Review the diff against scope (3.4). Commit at the gate (3.6). If the feature involves any non-deterministic behavior, add an eval (3.7); if it touches untrusted input or auth, add a security pass (3.8). Update all harness state and clear (3.9). Every artifact should exist and be honest: not performed for the checkpoint, but the genuine trail of how the feature got built.

Passing means you can produce, on demand, the full artifact trail for a real change: a spec you wrote before the code, a gate that actually ran, a scoped and reviewed diff, gate-boundary commits, and durable state a fresh session can boot from. Both capstones (Parts 4 and 5) are this same run repeated at scale, dispatch after dispatch, so an unpassed checkpoint means the capstones will be built on discipline you have not yet proven you have.

Sources