4.10 · Observability and Operations

4.10 · 20 min

Observability and Operations

At 2:14am, a customer's approval does not go through. You find out at 9am from an angry email, spend forty minutes reconstructing what happened from memory and guesswork, and never fully learn whether it was a one-off or the tip of something. The alternative costs you an afternoon now: structured logs that carry enough context to answer "what happened to this org at 2:14am" with a query, a handful of numbers you actually watch, and a runbook so the 9am version of you is following steps instead of improvising. Observability is not a dashboard you admire; it is the difference between knowing your product is broken before your customer does and finding out after.

Structured logging with context

The logging that helps at 2am is structured (machine-queryable, not prose) and carries context (which org, which user, which request). A console.log("upload failed") tells you something broke; a structured line tells you whose upload, in which org, on which request, with what error, so you can find every related event without grepping strings. The abstraction is one small module so the choice of sink is one file:

briefcase/lib/log.tsts
type LogContext = {
requestId?: string
orgId?: string
userId?: string
[key: string]: unknown
}

// Structured, context-carrying log. JSON lines are queryable in any
// log sink (Vercel, or a provider); prose is not.
export function log(
level: 'info' | 'warn' | 'error',
event: string,
ctx: LogContext = {},
) {
console[level](JSON.stringify({
  level,
  event,          // a stable name, e.g. 'approval.recorded'
  ts: new Date().toISOString(),
  ...ctx,
}))
}

Used at the seams, it turns an incident into a query:

briefcase/app/api/jobs/run/route.ts (excerpt)ts
try {
await handlers[job.kind](job.payload)
log('info', 'job.succeeded',
  { jobId: job.id, kind: job.kind, orgId: job.payload.orgId })
} catch (err) {
log('error', 'job.failed', {
  jobId: job.id, kind: job.kind, orgId: job.payload.orgId,
  attempt: job.attempts + 1, error: String(err),
})
throw err
}

Now "what happened to org X at 2:14am" is a filter on orgId across a time window, and every job, webhook, and request that touched that org is one query away. The requestId (generate one per incoming request, thread it through) ties a user's single action to every log line it produced, including the async job it spawned.

Error tracking on top

Logs are for querying after you know roughly what to look for; error tracking is for being told, unprompted, that something threw. For a Next.js app the current standard is Sentry, whose wizard sets up an instrumentation.ts that registers server and edge configs and an onRequestError hook, so uncaught exceptions in server components, route handlers, and jobs surface with stack traces and the request context. The install is a wizard run (npx @sentry/wizard@latest -i nextjs as of July 2026) that writes the config files; you verify it by throwing a test error and seeing it appear. Any equivalent (an error tracker that captures Next.js server errors with context) is fine; the requirement is that an unhandled exception in production reaches you as an alert, not as a customer email. Set it up in test first, confirm a deliberate throw lands in the dashboard, then trust it, the same prove-the-gate move as everywhere else.

The numbers for v1

You do not need a wall of graphs; you need a few numbers whose movement means something, watched consistently:

  • Error rate. The fraction of requests and jobs that fail. A step change is your earliest signal that a deploy broke something. Vercel's Observability tab surfaces function error rate and p95 with zero setup, which is enough to start.
  • p95 latency. The slow tail, not the average. Averages hide the users having a bad time; p95 is the honest "how bad is it for the unlucky" number.
  • Job failures / dead-letter depth. How many jobs landed in dead (4.8). A growing dead-letter count means work is being lost silently to the user even while every request looks fine, which is exactly the failure background jobs are prone to. This is a SQL query against your own jobs table, and it belongs on your watch list.

CURRICULUM notes two more if you later build the RAG feature (elective A8): AI cost per day and eval pass rate. You do not have that feature in Capstone A, so those two are not on the v1 list; add them if and when A8 ships. The discipline is to watch the few numbers that map to real failure, not to collect metrics because they are collectable.

The runbook

A runbook is the document that turns a 2am incident from improvisation into procedure. It does not need to be long; it needs to exist and be honest about the handful of things that actually go wrong. Briefcase's starting runbook:

briefcase/docs/RUNBOOK.mdmarkdown
# Briefcase runbook

## Where things are
- Logs: Vercel dashboard > project > Logs (filter by orgId/requestId).
- Errors: Sentry project 'briefcase'.
- DB: Supabase project 'briefcase-prod'. Jobs, subscriptions,
webhook_events are the tables you will actually query in an
incident.

## Symptom: a customer says an action failed
1. Get their org id (Supabase: orgs by slug/email of a member).
2. Query logs filtered to that orgId around the reported time.
3. Query jobs where org_id = X order by created_at desc: look for
 'dead' or repeatedly 'failed' rows; last_error tells you why.

## Symptom: billing looks wrong for an org
1. subscriptions row for the org: status + current_period_end.
2. webhook_events: was the relevant Stripe event received and
 processed_at set? A received-but-unprocessed event is the bug.
3. Cross-check the Stripe dashboard for that customer. The webhook
 is truth (4.7); if the table and Stripe disagree, a webhook was
 missed or errored: replay it from the Stripe dashboard.

## Symptom: digests did not go out
1. Check the enqueue cron ran (Vercel > cron logs, 13:00 UTC).
2. jobs where kind = 'notification_digest' for today: pending
 (worker not draining), dead (handler failing), or absent
 (enqueue cron did not run).

## Escalation / safe state
- If the worker is failing every job: disable the cron in Vercel
(stops the bleeding), jobs stay 'pending' and safe, fix, re-enable.
- Never delete rows to 'fix' state; jobs and webhook_events are the
audit trail of what the system did.

The runbook is a living document: every real incident either follows it (good) or reveals a gap you add to it (better). It is the operational twin of STATUS.md, and like STATUS.md its value is entirely in being kept honest.

Claude Code as a diagnosis partner

Production logs are exactly the kind of large, structured, boring data an agent reads faster than you. Point Claude Code at exported logs with a real question and it becomes a genuine diagnostic partner, generating hypotheses, correlating events, spotting the pattern in a thousand JSON lines. The discipline from 3.5 holds: it assists the reasoning, it does not own it. You bring the question and gate the hypotheses; it does the reading.

PromptLog diagnosis with an agent

Here are Briefcase's production logs for org_7f3 between 02:00 and 02:30 UTC (JSON lines, one event per line). A customer reports an approval that did not save around 02:14.

Do not speculate beyond the logs. Build a timeline of every event for this org in the window. Identify the first anomaly (an error, a missing expected event, an out-of-order sequence). State the single cheapest next check that would confirm or rule out your top hypothesis. Do not propose a fix yet.

[paste exported log lines]

That prompt keeps the agent inside 3.5's loop: timeline, anomaly, cheapest discriminating check, no leaping to a fix. It reads the thousand lines; you decide what the anomaly means and what to do.

Knowledge check

Knowledge check

Q1You log console.log('upload failed') at the seam where uploads break. At 2am a customer reports a failed upload. Why does this line not help, and what would?
Q2For Briefcase v1, this lesson says to watch error rate, p95 latency, and job/dead-letter depth, but NOT AI cost per day or eval pass rate. Why the exclusion?
Q3A customer's billing status looks wrong. Following the runbook, you find the subscriptions row disagrees with the Stripe dashboard. What does the runbook tell you to check, and why is that the right first move?
Q4You paste 1,000 lines of production logs into Claude Code and ask it to diagnose a 2am incident. What framing keeps this inside 3.5's debugging discipline rather than outsourcing your reasoning?

Checkpoint 4: the capstone gate

CheckpointCheckpoint 4 (Capstone A gate): Briefcase live and auditable

This is the Capstone A gate, and passing it means something specific and sellable: "production ready" is literally demonstrable to a client, not asserted. Not "the code looks done" but "here is the live URL, here is the tenancy suite proving isolation, here is the billing test surviving a failed payment, here is the job that dead-letters instead of vanishing, here is CI refusing my own broken PR."

Run the whole thing as one integrative check. Deploy is live. Open the three boundary suites (tenancy, billing, jobs) and confirm each includes its negative cases and passes in CI. Push a deliberately failing test on a branch and confirm the merge is blocked. Then do the human demonstration: on the live URL, sign up as a new agency, create a project, upload a deliverable, send it for review, approve it, and see the attributed audit record. That sequence is the product's core promise, working in production, on infrastructure whose failure modes you can prove you have handled.

Passing means you can now be paid to build and ship a production SaaS with an agent, because you can demonstrate the properties that separate a demo from a product: isolation that holds under attack, billing that survives failure, work that is never silently lost, and a merge gate that stops your own worst 1am instincts. An unpassed checkpoint means one of those properties is unproven, and unproven isolation or billing is the kind of gap that becomes a customer's data in the wrong hands or a charge you cannot account for. Do not treat this as done until every rubric line is demonstrable on the live instance.

Sources