Formation

"I'd add a semantic cache to cut costs." You just proposed serving wrong answers quickly, and you may not have noticed. Two different things get called caching in an AI system, and only one of them is free. Prefix caching. Your system prompt, tool definitions, and examples are byte-identical on every request and sit at the front of the prompt. That work does not need redoing. Exact match, deterministic key, zero correctness risk, and it is where most of the savings live. It does come with a real design constraint: the stable parts have to stay at the front and stay stable. Shuffle them per request and you throw the cache away. Semantic caching. Serving a stored answer to a question that is merely close. "What's our refund policy for EU customers" and "what's our refund policy" are semantically similar. They have different answers. When you pick a similarity threshold, you are picking a wrong-answer rate. That is a product decision, not an infra one. It can be defensible. Four conditions, and you should name all four: high volume, low stakes, a tight threshold, and a per-tenant namespace with a TTL tied to how fast the underlying truth moves. Skip prefix caching and jump straight to semantic and an interviewer learns you have drawn this system but not run it. → https://fm.dev/46pqhzr

A one-million-token context window is not a feature. It is a capacity number. Here is why. Every decode step needs to see every token before it. Recomputing that each time would be quadratic, so the state gets held in memory — for the entire duration of the generation, per request. That memory is finite. And shared. So: ten concurrent requests at 100K tokens each is a completely different machine profile than a thousand requests at 1K tokens, even if total token throughput is identical. Memory you spend holding one request's context is memory you cannot use to serve another request. Long contexts lower your concurrency ceiling. If that sounds familiar, it should. It is connection pool math. A finite pooled resource, consumed per request, held for the duration of the request. You have reasoned about this before. And when it runs out, you do not get a clean error. Requests queue. Some get evicted and recomputed. What you see is tail latency with healthy-looking dashboards — which is why nobody diagnoses this one on the first try. Which brings the week back to last week: "just put more in the prompt" is a capacity decision. It is the reason your retrieval step should return the right three chunks instead of the plausible thirty. → https://fm.dev/3TzDaEd

"I'd summarize the repo first" is an AI system design answer that sounds senior and usually isn't. You're designing an AI coding assistant. You have a 100 MB repo and your agent's context window only fits about 3 MB. Pre-summarizing sounds nice, but it costs ninety dollars a pass, goes stale on the next commit, and still sends you back to the code for anything specific. Episode 2 of Design Decisions shows the smaller starting point that holds up, and what to say when you're asked to defend it.

"It responds in about three seconds." That is not a latency answer. A streamed response does not have one latency. It has two, and they have different causes. Time to first token. When the user stops staring at nothing. Driven by prefill, so it scales with how long your prompt is. Time per output token. Whether it streams faster than they read. Driven by decode, so it degrades when the machine is busy. Three seconds total with a 300ms first token is a good experience. Three seconds total where 2.8 of them are silence and then a wall of text arrives is a bad one. Identical number. Here is the part that gets scored: they trade against each other. Bigger batches improve throughput and per-token pace, and make each individual first token wait longer. Picking a batching policy is picking which number matters — and only the product can tell you that. Chat UI: first token is everything. Overnight summarization job: nobody is watching, batch it hard, optimize cost. Agent making eight sequential calls: no human reads anything in between, so first token barely matters and the total budget is what you are dividing up. Give two targets. Give them as percentiles. Then say what happens when you miss — which is harder than it sounds, because a timeout on a streaming call has to deal with output the user has already seen. → https://fm.dev/4rcB1dV

Why does your LLM endpoint get slower when someone else sends a long prompt? Not because of traffic. Because you are running two workloads with opposite appetites on the same hardware. Every inference request has two phases. Prefill reads the entire prompt and produces the first token. All those tokens process in parallel, so it saturates compute. Decode generates every token after that, one at a time, each step re-reading everything before it. Almost no parallelism per request. It is bound by memory bandwidth, not compute. Opposite bottlenecks. Same box. Three things fall out of this, and they explain most of the weird behaviour: Batching is the entire throughput story for decode. One request decoding alone leaves the hardware idle, so serving systems keep rebuilding the batch continuously instead of waiting for fixed groups. A single 100K-token prompt hogs the machine during prefill and stalls everyone else's decode. Your p99 degrades because of another tenant's input length. That is a noisy neighbour, and you already know that problem. Input and output tokens are priced differently because they are genuinely different work. You do not need to have written a serving engine. You need to stop saying "the LLM call" as though it were one thing. → https://fm.dev/4xU06Nf

Most "use Redis" answers in system design interviews are one step ahead of the question. A hotel room hold is the classic case: ten minutes, one row, one expiry timestamp. A database already does this. Redis earns its place only when a specific thing starts to hurt — and knowing what that thing is is the difference between naming a technology and making a design decision. Episode 1 of Design Decisions breaks it down in 2 minutes, including what to actually say in an interview.

Everyone is now calling verification the new core engineering skill. Almost nobody has published an actual procedure for it. Here's the framing that makes it work: treat AI-generated code like output from a fast teammate whose work you haven't checked. Not an oracle. Not a toy. Usually close — occasionally wrong in ways that look completely fine. Reviewing it well is a repeatable loop, not a vibe. Step one: review the highest-risk areas first. Algorithm shape, loop boundaries, and whether the code actually matches the requirement. Not style. Style is the most visible layer and the least likely to hurt you. Step two: test against a real case list. Happy path, boundaries, equivalence classes, adversarial input. And a trap worth naming: generating test code and verifying correctness are different activities. Asking the model for tests is not the same as knowing the code works. Step three: choose your repair mode. This is the judgment call most people get wrong. Small bugs — a wrong index bound, a missing empty-input guard — fix by hand. It's faster, and it builds the understanding you'll need to defend the code. Major logic problems — don't patch piecemeal. Re-prompt with four things: the exact failing test case, observed versus expected output, the intended algorithm, and the complexity requirement. All four, not two. Underneath all of it: ask for one bounded piece at a time. Decomposition is what makes code reviewable — from models and from humans. And if AI is banned in your interviews, this loop still is the thing being graded. Testing assumptions and reasoning about boundary cases is what interviewers were always evaluating. Full write-up with the loop as a one-page checklist: → https://fm.dev/4irEAKX

AI-assisted interview rounds have moved from experiment to pilot at major companies — including formats where you're handed an AI assistant and evaluated partly on how you use it. At the same time, the consensus is that AI has eroded the signal from traditional coding rounds. Both facts point at the same conclusion: the question changed from "can you solve it" to "can you direct, explain, and verify it." When a model produces working code in seconds, typing speed and memorized syntax stop separating anyone. So here's what replaced them. Clarifying questions became the whiteboard skill. Interviewers are watching whether you interrogate an ambiguous requirement before writing anything. Control is what's being scored. Did you choose the algorithm and the architecture? Can you defend the trade-offs, even when the model typed the implementation? A candidate who accepts a working solution they can't justify scores badly — working solution notwithstanding. Confident and wrong is the failure mode. AI-generated code and explanations read authoritatively while being incomplete or incorrect. Complexity claims especially: frequently asserted, rarely verified. You have to actively check edge cases and hidden assumptions. The working pattern that reads well is small and reviewable. State your intent. Request a bounded piece. Read it. Test it. Then continue. The opposite — asking for the whole solution and hoping — is visible from across the room. Here's the part that should be reassuring: judgment about data structures, trade-offs, and edge cases transfers completely. Rote implementation practice matters much less than it did. And if your target company bans AI in interviews entirely, the habits are identical. Testing assumptions and reasoning about boundary cases is what was always being evaluated. → https://fm.dev/45K83bJ

Twelve months ago, "design a system that serves an LLM" was an ML-role question. It's now showing up in general software engineering loops, and at AI-first companies it can be an entire round. Candidates are responding by cramming transformer architecture. That's preparation for a question nobody is asking. When an interviewer asks you to design an AI-powered feature, they're testing system design judgment. The AI is a dependency — expensive, slow, non-deterministic, and occasionally wrong. Everything being scored is about how your system behaves around that dependency. Which means it's a distributed systems problem in a new shape. You already know how to reason about it. Four things earn points: Latency. LLM calls run an order of magnitude slower than a normal API call. What's your timeout? What does the user see while waiting — streaming or blocking? Cost. Token cost is per-request and non-trivial. When do you cache, and what's the cache key when the input is natural language? When do you route to a smaller model? Failure. The model returns nothing, returns slowly, or returns something confidently wrong. Those are three separate failures. Name them separately. What degrades, and what's the fallback path? The correctness boundary. What in this system is allowed to depend on model output being right, and what has to be verified before it reaches a user? Talk about RAG as retrieval and generation — two stages, each with its own latency and failure budget — not as a lecture on embeddings. And the failure modes seniors are expected to raise unprompted: circuit breakers around the provider, graceful degradation to a non-AI path, and a monitoring story for output quality, not just uptime. What to actually study is a short list, and it's mostly things you already know. → https://fm.dev/3SukRja

"It's AP." Two words, and the interviewer already knows how deep your understanding goes. CAP is the most-cited and least-understood idea in a system design interview. Three specific misunderstandings show up over and over, and each one produces a worse answer than saying nothing at all. Availability does not mean "the data is there." In CAP, availability means only that the system returns a successful, non-error response. It says nothing about whether that response is current. An available system will hand you a stale account balance with total confidence. CAP consistency is not ACID consistency. CAP consistency is about what values reads can return across replicas. ACID consistency is about preserving invariants inside a transaction. Same word, unrelated concepts — and conflating them is a visible tell. CAP is not a permanent "pick two." It describes the trade-off during a network partition, not a standing property of your system. Partitions are effectively unavoidable, which is exactly what makes this a practical design question instead of a theoretical one. Here's what the trade-off actually looks like when it's concrete: A bank balance should reject requests rather than risk a double-spend. A shopping cart or a likes counter should keep accepting updates and reconcile conflicts later. Same architecture, opposite call — and the reason is the business consequence of being wrong. So instead of "it's AP," try the sentence that shows you've made a decision: "During a partition, this subsystem should reject writes, because the cost of a stale read here is X." And the part that reads as senior: one product usually needs both. Naming which subsystem gets which is the move. Full breakdown, including what to say when the follow-up comes: → https://fm.dev/4qpPVxf