CS Survival Kit

Your automation runs perfectly for three weeks, then charges a customer twice on a Tuesday. Nothing changed. Nothing errored. If that sentence gives you a specific feeling in your stomach, this lesson is about converting the feeling into diagnoses. You do not need a computer science degree to operate agentic systems, but you need about five concepts at intuition level, because they are where systems break in ways that do not look like breakage. No proofs. Scenarios only.

State: the question "where does it live?"

Scenario. Your follow-up bot tracks which leads it already messaged so nobody gets double-texted. It works flawlessly in testing. In production it randomly re-messages people, but only after deploys.

The concept: state is any remembered fact that affects behavior. The counter in P.6's closure, a variable in your script, rows in SQLite, the "already messaged" list. The most important question about any piece of state is where it lives, because location determines lifetime. State in a variable dies with the process. State in a file survives restarts but lives on one machine. State in a database survives everything and is shared. The bot above kept its list in memory; every deploy restarted the process, the list was reborn empty, and the bot's memory was wiped clean without a single error. The fix is not better code, it is relocating the state to storage that outlives the process.

When any system "forgets" something, ask where that fact lived and what resets that location. Deploys, restarts, crashes, and autoscaling all execute the same attack: they kill processes. Memory-resident state is a bet that the process lives forever, and the house always wins.

Concurrency: two things touching one thing

Scenario. Your billing webhook fires twice for the same event (networks retry; this is normal and guaranteed to happen eventually). Both copies run your handler at the same time. The handler reads the customer's balance (100), adds the charge (25), writes back the total. Both copies read 100 before either wrote. Both write 125. The customer paid 50 and their balance says 125.

The concept: concurrency problems are almost always two things touching one thing: two processes, one bank balance; two webhook deliveries, one order row. The read-modify-write pattern is the classic victim, because the three steps of one actor can interleave with the three steps of another. The symptoms are maddening by design: rare, timing-dependent, impossible to reproduce on demand, gone when you add logging.

You need the smell, not the theory. When something impossible happened once and never again, ask: what shared thing were two actors touching, and could their steps interleave? The fixes have names you will meet in Track A (transactions, idempotency keys, unique constraints), and one you already know: the database UNIQUE constraint from P.7 is a concurrency defense, because the database itself refuses the second insert no matter how the timing landed. An idempotent handler, the property the course will demand of every webhook you ship, is one where running twice has the same effect as running once, which converts "guaranteed eventual double-delivery" from a catastrophe into a shrug.

Complexity: why the nested loop over 29k leads hurts

Scenario. An agent writes you a dedupe script: for every lead, scan every other lead for a matching email. Demo data, 200 leads: instant. Production, 29,000 leads: you kill it after twenty minutes, assuming it hung. It did not hang. It was working through 841 million comparisons.

The concept: what matters about an algorithm's cost is how it grows as data grows. Compare-each-to-all grows with the square of the input: 200 leads means 40 thousand comparisons, 29k leads means 841 million, a 145x increase in data producing a 21,000x increase in work. Nothing "broke": the code that was instant in the demo was the same code that was unusable in production, and no error will ever tell you so. The fix pattern is almost always the same: one pass to build a lookup (emails seen so far, in a set or map, or that database index from P.7), one pass to check against it. Two passes instead of a square.

The smell to install: a loop inside a loop over the same large collection. When you see it in generated code, ask one question: how big can these collections get in production? Ten items, fine, move on. Thousands, flag it. Demo-sized data is how quadratic code passes review, and agents produce quadratic code readily because it is the simplest correct-looking answer.

Caching: a copy that can lie

Scenario. To stop hammering your CRM's API, someone caches the contact list for five minutes. API costs drop 90%. Then sales complains: a lead updated their phone number, the rep called the old number, the lead had already been called twice by then. Both computations were "correct." The cache was serving the past, as instructed.

The concept: a cache is a copy of expensive-to-get data kept close for cheap rereading. The win is real and often enormous. The cost is a new failure mode that did not exist before the optimization: staleness, the copy disagreeing with the truth. Every cache decision is the same tradeoff: how stale can this particular data be before someone gets hurt? Stock prices: seconds. A country list: a year. There is no universally right answer, which is why "add caching" is never a complete instruction, and why the second-order question (how does it get invalidated when truth changes?) is the one that separates operators from optimists. When data seems wrong but flickers right, or wrongness lasts exactly N minutes, suspect a cache between you and the truth.

Queues: the shock absorber

Scenario. Your intake form calls three APIs, sends two emails, and writes a row before responding. Under normal load, four seconds. During a webinar spike, forty-five, and browsers time out, and users resubmit, and now you have duplicates too (hello, concurrency).

The concept: a queue decouples receiving work from doing it. The form handler does the minimum (validate, write one row, enqueue a job, respond in 200ms), and a separate worker chews through jobs at whatever pace it sustains. Spikes become backlog instead of failures; backlog drains. The price: the work is now asynchronous, meaning "submitted" no longer implies "done," and your system needs answers for "how far behind are we?" (queue depth, a number worth watching) and "what happens when a job fails?" (retries, and after enough retries a dead-letter pile a human looks at). You have operated queues without the name: every Zapier task history with its pending and replayed tasks is one. Part 4 has you building them.

Tutor drill

Run it. This lesson is quiz-heavy by design: intuition is built by diagnosing, not by reading about diagnosing.

Where it breaks

The trap with vocabulary is wielding it as incantation. "We should add caching" and "let's make it a queue" are optimizations with invoices attached: caching buys speed with staleness, queues buy resilience with asynchrony, indexes buy reads with writes. The professional version of each sentence includes the payment: "cache contacts for five minutes, and here is why five minutes of staleness is safe for this data." When an agent proposes one of these (and agents love proposing all three), the review question is always the same: what does this cost, and did anyone decide we can afford it?

Knowledge check

Sources

Nothing in this lesson is version-sensitive: state, interleaving, growth rates, staleness, and backlog are properties of systems, not of any tool's release notes. The scenarios are composites of failures every operator eventually meets; the lab for this lesson is your next production incident, which will arrive on its own schedule.