How we built a Slack-native on-call reasoning harness at Instacart that shortens time to first insight, speeds up theory testing, and turns tribal knowledge into reusable infrastructure.

Key Contributors: Karthik Halukurike, Gabe de Oliveira, Hassan Jallad, Alan Wong

On-call work is a race to turn noisy signals into shared judgment. The hardest minutes of that race aren’t the ones spent fixing the bug — they’re the ones spent figuring out what the bug even is, while everyone in the thread is asking the same question from a slightly different angle.

Blueberry is the system we built at Instacart for those several minutes. It lives in the Slack thread where the team is already coordinating, picks up each alert as it fires, and lands a grounded explanation back in the thread in about three minutes — thousands of times a month.

On-call is mostly a clarity and speed problem

Most on-call pain does not start with a major incident. It starts in the noisy window right after a page fires, when the thread fills with links, dashboards, guesses, and partial context. Someone asks whether a deployment caused it. Someone else asks how broad it is. A third engineer joins and asks what is going on.

In that moment, the hardest problem is often not deep root-cause analysis. It is getting to a shared understanding quickly enough that the team can make a good next move. For us, that made two metrics matter most: time to first insight (TTFI) and time to test theories (TTTT). TTFI comes from auto-triaging every qualifying alert the moment it lands. TTTT comes from engineer-initiated on-demand investigations and follow-ups as the investigation deepens.

Blueberry is a Slack-native on-call reasoning harness

Blueberry was built for that first window of confusion. Blueberry lives in Slack because that is where on-call engineers already coordinate, ask clarifying questions, and make decisions under pressure. We wanted the system to work inside the shared operational conversation so the full team stays in sync in real-time and moves faster.

The right mental model for Blueberry is a reasoning harness: one shared runtime that can load context, expose the right evidence surfaces, apply team-specific overlays, and produce a grounded response in the same thread where the work is happening.

Architecture: a durable, tool-aware reasoning harness

The harness is a small number of components arranged so that any single diagnostic pass is durable, tool-aware, team-aware, and inspectable. The shape matters more than the parts, because most of what makes Blueberry sharp comes from how these pieces interact during a diagnostic pass rather than from any one of them in isolation.

Ingress is selective. Blueberry watches Slack thread events and only promotes the ones that deserve a diagnostic pass: auto-triggered alerts, @mentions, DMs, follow-ups in an active thread. The balance is between responding to too much or too little.

The job management layer is durable, not best-effort. Diagnostic passes are enqueued onto a Postgres-backed durable queue so that they survive worker bounces, restarts, and takeovers. Multiple worker threads can drain the same queue in parallel. If a worker fails mid-pass, another picks the work up cleanly from the persisted state. For an on-call system that has to keep responding at 3am during a rolling deployment, this is a load-bearing decision.

The processor is the composition layer. Before the agent loop starts, the processor resolves which team owns the channel, loads the team profile, layers in team-specific skills and MCP endpoints on top of the shared baseline, and prepares the tool surface that the agent will be given. Specialization happens here, not inside the agent. From the agent’s perspective, the right tools are simply present.

There are three Model Context Protocol surfaces, not one. An in-process MCP runs alongside the agent for fast local helpers that share session state. A standalone shared MCP server hosts the heavier shared investigators, such as deploy and flag analysis, service-dependency lookup, error analysis, and anomaly sweeps, so they can be scaled and updated independently from the agent runtime. Team-hosted MCPs are mounted remotely so that data sources unique to a team show up as first-class tools without being baked into the core. The three surfaces have different latency, sharing, and ownership properties, and the processor presents them to the agent as one tool catalog.

A single diagnostic pass can fan out into parallel sub-agents. The agent loop can spawn focused sub-agents. The alert-investigate skill uses this to launch internal knowledge searches over postmortems, runbooks, and prior incidents in parallel with live signal collection, change collection, and operational context collection. Wall-clock time is set by the slowest collector, not the sum.

Persistence captures reasoning, not just outputs. Every diagnostic pass writes both a session-context snapshot, so the next pass or a different worker can take over without losing thread state, and a reasoning trace, so developers can later inspect how the agent decided what to do. These artifacts are what make the rest of the learning loop possible.

Reasoning ladder: where the agent gets freedom, where we hold the line

For an on-call agent, the meaningful design question is not just which model to use. It is where the system should let the agent reason dynamically, and where the harness should hold a hard line for reliability.

An earlier implementation used a more workflow-shaped agent hierarchy, which was predictable but made it hard for the system to choose what to do next when an alert did not match the expected path. Moving to a more dynamic agent loop let Blueberry decide when to call tools, fan out to sub-agents, deepen into evidence, or stop and respond, while the harness kept reliability constraints around persistence, output structure, and evidence discipline.

The interesting part is the balance of freedom and structure. The model is best at choosing what to look at, weighing evidence, and deciding when an investigation is done. The harness is best at the parts that have to stay stable across thousands of diagnostic passes like capturing reasoning, keeping output predictable, and enforcing the discipline that makes the synthesis trustworthy.

From alert to first insight

Alerts do not arrive in one clean format. They show up as monitor links, exception tracker issues, paging alerts, or raw text copied into Slack. Blueberry starts by normalizing those inputs into a shared context, optionally enriching them with team information, and then gathering evidence in parallel instead of one source at a time.

That parallelism is important. Historical context matters, so Blueberry looks for postmortems, runbooks, and prior incidents. Live operational evidence matters too, so it checks signals, changes, and surrounding context at the same time. The goal is to move from “something fired” to a first grounded explanation as quickly as possible. Across roughly 17k auto-triggered triages in April 2026, TTFI averaged about 3 minutes.

Example: SEV2 before declaration

Blueberry was automatically triaging a wave of E2E test failures in a staging environment. Before the team had opened an incident channel, it identified the exact root cause: a code change had been deployed ahead of its corresponding database migration. The ORM threw a runtime error when it tried to access an undeclared enum attribute in the orders domain. Blueberry surfaced the full call stack, the deployment sequence, the dependency graph, and a clear explanation of what happened.

Minutes later, the team declared the incident. The timeline below shows the important point: the first grounded explanation landed before the incident room fully formed.

This is the scenario Blueberry was built for: the window between the first signal and the moment the team assembles. Getting a grounded answer into that window compresses the entire response timeline, from the first alert through remediation.

From first insight to tested theories

Getting to a first explanation is only half the job. The next question is how fast the thread can move from speculation to evidence. Blueberry helps test whether the likely culprit is a deploy, a feature flag, a dependency regression, or a recurring failure pattern that already exists in historical context.

The design here is deliberate. Historical priors should inform the investigation, but live evidence should decide it. Blueberry compares the two, then narrows the next move: inspect a suspicious PR, review a flag change, look at a dependency edge, or go deeper into a trace or stack pattern if the first pass points in that direction. Across roughly 7k engineer-initiated theory checks and follow-ups in April, TTTT averaged about 3 minutes — about the same shape as TTFI on the original alert.

This is where the loop matters most. The agent can inspect the proposed culprit, test adjacent evidence, revise when the engineer adds context, and keep narrowing until the remaining explanation is backed by logs or code.

Example: 8-turn PR exoneration ending at a smoking-gun WARN log

An engineer walked into a thread worried that his PR had broken staging notifications. Over eight turns, Blueberry worked through the evidence with him. It first confirmed the PR did what reviewers had asked: it converted an IllegalArgumentException to a typed NotFoundException and removed a catch-all Status.INTERNAL handler. When the pipeline failed, Blueberry traced the failure to a byte-diff between two markdown files the PR does not touch and pointed out that master was already broken pre-merge.

The engineer asked whether a separate Slack report from a parallel team was related. Blueberry reframed the investigation around what could plausibly produce the reported behavior, based on what the PR actually changed. It then pulled UpdateConfig calls from mesh logs and found that the only staging error occurred three minutes after the original report, which made it timing-incompatible with the trigger.

The smoking gun came from backend logs: a continuous WARN, “The event doesn’t exist or is not configured for notifications: updateDeviceConfig,” which had been firing for roughly 34 hours before the merge. Blueberry explained the design: the staging service config had updateDeviceConfig: [], an empty subscriber list by design, so the notification framework correctly logged WARN and returned before posting to Slack. It then drew the distinction in code: updateStoreConfig had subscribers and posted, while updateDeviceConfig had an empty list and stayed silent, both in ConfigManagementService.kt.

The engineer had code-level proof that the gap was a pre-existing configuration, not his change. This is the long-form, multi-turn investigative arc Blueberry is designed to support: not a one-shot answer, but a careful examination backed by code and logs.

Example: a recurring external outage caught by historical pattern

When error rates appeared to be elevated from an external integration, the instinct was to look for a recent code change. Blueberry checked recent deployments, found nothing suspicious, and then cross-referenced against historical retailer context. It surfaced two prior incidents with the same error signature and the same root cause: a partner-side outage, not Instacart’s code. The investigation pointed directly to the retailer’s escalation path. Time from alert to actionable conclusion: under 8 minutes.

The two-pass synthesis was designed for this. The historical match did not boost confidence on its own; it was confirmed by the absence of contradicting live evidence on the deploy side, which let the agent flip from “probable internal regression” to “external dependency” without overshooting.

Why specialization matters more than generality

A fully general agent can sound flexible while still being too broad to feel sharp in production. Blueberry became more useful as we made specialization a first-class design choice. The shared harness still works even without a team-specific setup, but it gets significantly better when teams contribute their own profile notes, skills, and MCP endpoints.

By April 2026 in the early rollout, roughly 55% of teams had onboarded team profiles. The important part is not just that more teams were present; it is that their local habits, runbooks, skills, and MCPs were available to the same base harness, so the system could stay general in architecture while becoming local in practice.

That is where tribal knowledge starts to become reusable infrastructure. The little patterns that usually live in experienced engineers’ heads, the “if you see X, check Y” habits, the team-specific data sources, and the known failure modes all become things the harness can reason across in a repeatable way.

Slack-native collaboration is part of the product

Blueberry is useful partly because it improves the shared conversation, encourages collaboration and extracts tribal knowledge into the shared space. When the system clarifies likely scope, summarizes what changed, or explains why a theory looks weak, that understanding becomes visible to everyone in the thread at once.

That changes who can participate. Late joiners can become useful faster. Secondaries do not need to reconstruct the whole thread from scratch. And more of the team’s reasoning becomes widely useful, legible and reusable.

The most useful feedback signals were in the moments when engineers reacted to Blueberry’s sharp insights, speed of root-causing and ability to validate their theories rapidly

Because of the clear benefits of Blueberry, engineers are also investing in the tool, rapidly improving its effectiveness. Teams build skills, write runbooks in a format Blueberry can use, and push back when an investigation is wrong. They can engage with it in the way they engage with a capable colleague.

Better on-call hygiene, not just faster triage

On-call is partly about root-causing alerts and partly about proactively maintaining system health. Blueberry helps by keeping thread context legible across handoffs, reducing repeated reconstruction loops, and giving the team a shared memory of what has already been checked. That becomes most visible during alert storms, when the real problem is lack of convergence.

That same shared memory matters during primary handoffs too: a new on-call can see current state, tested ideas, likely downstream impact, and open questions without restarting from scratch.

The tool is now part of better on-call hygiene: fewer repeated reconstruction loops, cleaner shift changes, and a better shared memory of what the team already checked.

Example: 90 alerts, one root cause, 3 minutes

When a cluster-wide disruption hit the data engineering Kubernetes cluster, 90 P1 and P2 alerts fired across multiple namespaces in under 3 minutes, a cascade triggered by JobManagers going unhealthy simultaneously. Blueberry handled more than 50 investigations in parallel. Every single one arrived at the same conclusion: a cluster-level infrastructure failure, not application code, not a feature flag, not a bad deployment.

No engineer had to triage all the alerts. The shared thread became a single point of ground truth before anyone had assembled the incident team. This is also the kind of scenario that pays back the architecture choices directly: a durable job queue, parallel workers, three composable MCP surfaces, and a processor that can produce consistent synthesis at fan-out scale.

What we are testing next

The next frontier is closing the loop between diagnosis, safe action, and learning. Early work is already exercising deploy-failure mitigation: when an internal service deploy fails, Blueberry can gather canary signals, inspect logs and error context, correlate the failure with recent PRs, and decide whether the right move is to redeploy, escalate, or stop because the evidence is unclear.

The key is that action has to move through a safety path, not a shortcut. Before execution, Blueberry needs explicit preconditions, evidence analysis, an action plan, action-plan reflection, an execution policy check, and result reflection. In parallel, reasoning traces and self-assessments are feeding reviewed prompt, skill, and routing improvements. That loop is being tested and scaled as part of the same harness: diagnostic passes produce evidence, safe actions stay policy-gated, and outcomes become learning material.

What changes at the organizational level

As Blueberry became useful in real threads, it started changing behavior. Engineers saw the system helping in the flow of actual work, which made AI adoption feel practical instead of abstract. At the same time, teams had a reason to externalize their own operational knowledge — runbooks, skills, MCPs — in ways the harness could actually use, so the best debugging instincts could be applied consistently, by more of the team.

That is Blueberry’s force-multiplication story. It helps the primary on-call engineer in the moment and gives teams a way to package their best debugging instincts so they can be reused.

Conclusion: Useful AI systems in operations need the right harness

The biggest lesson from building Blueberry is that model quality matters, but harness quality decides whether an agent is actually useful in operations. Grounding matters. Specialization matters. Continuity matters. Collaboration matters. A learning loop matters. When those pieces come together, an agent stops feeling like a demo and starts feeling like real operational infrastructure.

What that looked like in production in a single month:

• Blueberry ran ~25k diagnostic passes across 270+ Slack channels, ~17k+ auto-triggered alert triages and ~7k+ engineer-initiated theory checks and follow-ups.
• TTFI averaged ~3 minutes across the auto-triggered triages
• TTTT averaged ~3 minutes across the engineer-initiated follow-ups.
• The system completed ~23k of those passes successfully with a 99.9% success rate.
• The same harness handled about 58k+ Model Context Protocol tool dispatches and supported ~60 onboarded team profiles, with sustained usage in dozens of channels.

Every on-call shift produces knowledge that can disappear. Blueberry is our attempt to make that knowledge durable. The diagnostic harness captures some of it. The team profiles, skills, and learning loop capture the rest. The best operational instincts now compound, instead of expire.

*Examples are illustrative and fictionalized based on real production patterns the agent handled

Blueberry: Force Multiplier For The On-Call Engineer was originally published in tech-at-instacart on Medium, where people are continuing the conversation by highlighting and responding to this story.

Sergio Camelo, Caitlin Kearns, Matias Cersosimo, and Tilman Drerup

As artificial intelligence increases the velocity of engineering and science teams, experimental throughput is set to become a bottleneck for many product decisions. Many companies can now build faster than they can experiment, with queues of good ideas running the risk of not being tested because of lack of experimental capacity.

This problem is particularly severe in marketplaces, where the presence of spillover and cannibalization effects between experimental units requires cluster-level randomization techniques. That randomization, in turn, has the unfortunate tendency to substantially reduce statistical power and slow down experimentation. In this post, we show that the predictability of outcomes at fine grains can be exploited to reduce the variance of aggregate metrics, even when experiments themselves are run at a coarse level. Since statistical power depends on metric variability, this yields considerable reductions in experimentation time.

The Interference Problem

In marketplace settings, behavior and outcomes for individual participants are inherently intertwined. In a delivery marketplace like Instacart, for example, the dispatch system solves a bipartite matching problem between shoppers and customer orders. Since assignments are global and interdependent, matching an order to one shopper means that the same order cannot be matched to another shopper. As a result, changing the handling for a single order creates ripples that affect the orders around it. If an experimenter were to assign a treatment intervention to one of these orders while leaving neighboring orders as controls, the latter would evidently be contaminated.

A common response to this problem is to randomize treatments at the level of a cluster, chosen so that interference can stay within it. In food and grocery delivery, that cluster is typically a geographical region. Since every order within a region sees the same treatment, contamination is contained at the regional boundary rather than leaking between treatment and control. The drawback of such region-level designs is that they give us far fewer units of randomization than unit-level designs, and fewer units lead to less statistical power.

One way to recover some of that power is a switchback design. With a switchback design, each region is re-randomized between the treatment and control group every fixed window (typically one day). The unit of analysis then becomes the region-day rather than the region, and the effective sample size grows with the duration of the experiment. This, of course, relies on the assumption that a window’s treatment effect is realized within that same window and does not spill into the next. Still, even with this adjusted design, the effective sample size typically remains small, and power remains an issue. At Instacart, running powered experiments may take months, and for very noisy metrics, may even be entirely infeasible in any reasonable timeframe.

Enter CUPED

One way to recover some of the lost power is CUPED (Deng, Xu, Kohavi, and Walker [2013]), layered on top of a switchback design. The intuition behind CUPED is that if the outcome metric has a component that can be predicted from pre-treatment features, then subtracting that predictable component removes noise without removing the treatment effect, thus making the effect easier to detect. Let’s make this concrete.

Throughout, subscript k indexes a region-day observation. Suppose we run a switchback experiment and we observe an outcome metric Yₖ for region-day k. Let Zₖ be a vector of pre-treatment features for that region-day, and let Xₖ = f(Zₖ) be a prediction of Yₖ built from those features, where f is fit on pre-treatment data. We form the CUPED-adjusted outcome:

This adjusted outcome has the same expectation as Yₖ, so it can be used to estimate the treatment effect without introducing bias, while its variance is reduced by a factor of 1 − ρ², where ρ is the correlation between Yₖ and Xₖ. The better Xₖ predicts the outcome, the larger the variance reduction, and therefore the shorter the experiment needed to detect a given effect size.

This naturally leads to the following question: How good can this predictor get? A key limitation of CUPED under interference is that the predictor Xₖ must be built at the coarsest level of randomization. At the region-day level, the features available — day of week, average basket size, shopper supply, weather — all capture broad patterns but miss the specific mix of orders that land in a region on a given day. However, for a metric like late-delivery percentage, most of the variation comes from order-level characteristics, e.g., where each delivery is going, what is in the cart, what window the customer chose. Averaging those away before building the predictor discards the signal that would make it useful. In many experiments, we have found that standard CUPED constrained to region-day predictors can cut variance by up to 10% after controlling for geo- and day-fixed effects. That is a good start, but not enough to meaningfully shorten experiment duration.

Order-Level CUPED

While we cannot experiment at the order level, we still have access to plenty of order-specific information that we know drives our metrics. If order-level outcomes are what we ultimately aggregate into our metrics, why not predict at that level and aggregate the predictions the same way? This keeps the structure of CUPED intact and allows us to take into account order-specific covariates by training a suitably-chosen machine learning model.

Concretely, our methodology works in two steps: First, we train an order-level model g that predicts each order’s outcome, ŷᵢ = g(zᵢ), using only features zᵢ available before the fulfillment system acts on the order. Second, we aggregate this model’s predictions up to the region-day grain using the same aggregation the metric itself uses (e.g., a mean, for a metric like late-delivery percentage):

The aggregated prediction Ŷₖ then becomes Xₖ in the CUPED adjustment from the previous section: it is a pre-treatment prediction of Yₖ, now informed by order-level detail that a region-day feature could not capture.

A Word of Caution

Before we go to the results, one word of caution is necessary to ensure that the expected variance reduction does not come with an increase in bias for the treatment effect. When estimating the first step regression, every feature used by the order-level model must be unaffected by the treatment. In predicting an order’s lateness, features like items in the cart, chosen delivery option, store location, time of day, and weather are safe because they are fixed before any fulfillment algorithm acts on the order. Features like the shopper assigned to the order or how many orders that shopper handles on the same trip are off the table because they are downstream of the treatment. The distinction matters because some of the forbidden features may be the best predictors of the outcome, and therefore the most tempting to add.

This assumption is essential, and we can test it. To do so, we run a mean difference test on Ŷₖ, the prediction built from pre-treatment features, in place of the real outcome Yₖ. Since Ŷₖ depends only on pre-treatment features, its estimated treatment effect should be statistically indistinguishable from zero. If it is not, something in the feature set is being moved by the treatment, and the variance-reduced estimate cannot be trusted for that experiment.

The plot below illustrates this in three experiments, reporting for each the variance reduction from order-level CUPED alongside the placebo test that flags feature contamination. In our marketplace, for example, we experiment with improved versions of the models that predict the ETA of an order, which affects the ETA shown to users before they place an order, so they can make an informed decision. This shifts the distribution of orders placed, and the placebo test correctly flags this as contamination.

Results: Experimentation Speed-Up

We tested the method across 10 experiments aimed at reducing order lateness, one of Instacart’s highest priorities. For each experiment, we trained a Gradient Boosted Tree to predict whether an order would be late. Our models fit the data fairly well, with expected features doing the heavy lifting in the prediction, including weather and basket size.

We then did a side-by-side comparison of standard CUPED with region-day features and order-level CUPED with order-level features. The gap is stark. Order-level CUPED cuts variance by 18 to 40%, more than 3 times the 5 to 10% reduction we get from only applying region-day CUPED.

These reductions in variance have immediate and notable effects on experimentation velocity, allowing us to cut experimental runtimes by about a third on average. This added velocity allows us to run more experiments per quarter and rapidly identify improvements in the algorithms that power our marketplace, including the models that ensure that customers get their groceries on time.

Wrapping Up

When the unit of randomization is coarser than the unit of observation, small effective sample sizes tend to slow down experimentation. In this post, we showed how building predictions at finer grains, in our case order-level, and aggregating them up can recover variance that standard region-day CUPED leaves on the table. The move is counterintuitive at first, since interference is exactly why we randomize coarsely and avoid order-level outcomes in the first place. The crux of our solution is that the predictions only ever enter through a pre-treatment covariate, leaving the estimates unbiased. As our results above show, this change can have meaningful effects on experimental power and velocity.

Importantly, the presented approach is not specific to the delivery context. In settings where interference forces randomization into coarse clusters (like with graph clusters in social networks, keyword groups in ad auctions, or geographic markets from lodging to dating), outcomes are still observed on a much finer unit than the one being randomized. That gap is exactly what the proposed method exploits, and its underlying idea should be directly applicable.

The Economics and Data Science Teams at Instacart frequently write about the methods we deploy to solve challenging marketplace problems. Recent posts include work on using regression discontinuity design in optimization problems and a perspective on the evolving role of artificial intelligence in the work for applied scientists. Follow tech-at-instacart to be notified as we release new posts.

Variance Reduction Below the Randomization Grain was originally published in tech-at-instacart on Medium, where people are continuing the conversation by highlighting and responding to this story.

Benjamin S. Knight

Scaling Marketplace experiments requires specialized statistical techniques. We examine why standard ordinary least squares regression (OLS) becomes computationally intractable when controlling for high-cardinality categories. We then dive into the underlying math and demonstrate how modern packages — specifically Fixest and Pyfixest — bypass these limitations. We conclude by benchmarking these methods to show their real-world impact on processing speed, memory efficiency, and estimator precision.

At Instacart we strive to give our customers access to all the fresh foods and ingredients that they would normally get from a trip to the grocery store, but without the hassle of driving, finding parking, waiting in line, etc. Instacart’s Marketplace team is responsible for surfacing customers’ orders to shoppers, aligning Instacart’s delivery windows with shoppers’ projected availabilities as efficiently as possible.

This entails a careful balancing act. If we offer delivery windows that are sooner / more popular, then we risk overextending shoppers’ ability to fulfill those orders on time. If we are too conservative in our delivery option offerings, then we risk losing potential orders. Accurately measuring the impact of changes in our batching and routing algorithms requires thoughtful experiment design and software.

Experimentation on Marketplace

One of our primary concerns in Marketplace is treatment spillage. For example, if we adjust our batching algorithm and increase the rate at which multiple orders are combined into batches in Brooklyn and Queens, then we face a real risk of also influencing the rate of batch creation / completion in Staten Island, the Bronx, and Manhattan. In this case the treatment impacts the control group — a classic source of measurement bias as a consequence of violating the Stable Unit Treatment Value Assumption (SUTVA). For this reason, we typically analyze experiments using a time-series crossover design like the Geo:Time Switchback Mode design below:

Unpacking the expression above, the unit of experimental assignment is tuples of geographic regions (‘geos’) and time periods (days, 4-hour time blocks, etc.). The measured outcome of some metric y in geo z at time t is a function of whether or not that geo:time pair was treated (i.e. the treatment variable), the unique intercept term for that geo, the unique intercept term for that date (this captures day of week seasonality, major holidays, etc.) and the error term for that geo:time. By defining the unit of experimental assignment in terms of both time and space, we capture a much richer set of data. This additional information allows us to effectvely flag and statistically adjust for treatment spillage.

There are also times when we need to be concerned about treatment spilling over across time units. For example, a shopper experiencing the treatment might accept so many batches that they decide to not work the subsequent day— regardless of whether the treatment is still active then. In these situations when the influence of the treatment persists, we need to run a Static-Geo experiment where we fix experimental assignment for that geographical unit for the entirety of the experiment.

The formulation is similar to the geo:time switchback framework, but with a key difference. Because we are dealing with geographic regions instead of geographic region:time slices, our effective sample size is an order of magnitude smaller. This significantly reduces the available statistical power. To help mitigate this, we can add data from before the beginning of the experiment. The interaction term then acts as a gatekeeper — subsetting the data to time units when the treatment was live.

Geo:time switchback and static-geo are two of the most common experiment designs on Marketplace. Both approaches leverage fixed effects to improve estimate precision. The use of fixed effects is especially important for static-geo experiments given that the geographic region will only ever experience the control OR treatment (not both unlike the switchback design).

The High-Cardinality Bottleneck

While the ability to apply fixed effects is critical for building performant models, we begin to run into problems when the number of groups being controlled for increases. Let’s revisit the solution for the 𝛽 coefficient within the OLS normal equations (most software does not actually use this approach, but it is a good starting point).

The right side — Xᵀy — is the dot product of our predictors X and the response variable, y. It is a relatively cheap operation — only O(nk) where k is the number of predictors and n is the number of observations. The left side (inside the parentheses) is the Gram matrix ( XᵀX). Taking the product of the feature space and its transpose is more expensive — O(nk²). Inverting the Gram matrix is where our problems begin. Matrix inversion is O(k³) in terms of computational complexity, i.e. a 2-fold increase in the number of predictors induces an 8-fold increase in the compute time.

This becomes a problem in the context of high-cardinality fixed effects, since each distinct level within the fixed effect requires its own dummy variable. For example, if we have 1,000 regions then our Gram matrix grows to encompass 999 additional dimensions. Taking the inverse of such a matrix would increase the required compute time nearly a billion times. If we do nothing, then more ambitious applications of fixed effects (e.g. shopper-level fixed effects, customer-level fixed effects) become intractable — both computationally and with respect to memory requirements.

The problem gets worse. As mentioned, most statistical software does not actually use the normal equations. For example, OLS estimation in Statsmodels defaults to Moore–Penrose inverse via singular value decomposition (Note — Scikitlearn also uses a SVD-based approach). This alternative methodology protects users from ‘silent’ (i.e. catastrophic) failures of the normal equations in the context of near-collinear predictors, but at the cost of increased time and memory footprint.

Admittedly, getting linear coefficient estimates from large feature sets is not a new problem. For many applications, stochastic gradient descent and other iterative methods provide viable estimates. At the same time, it would be ideal if there was a solution with zero stochastic component and no hyperparameters. In the next sections, we’ll walk through how we can achieve this through a combination of the Frisch-Waugh-Lovell theorem and the method of alternating projections as implemented in fixest / pyfixest.

Applying the Frisch-Waugh-Lovell Theorem

Gram matrices can only grow to a certain dimensionality before becoming practically uninvertible. Fortunately, we have a tool that allows us to slim down the Gram matrix to only contain the predictors we are interested in. First articulated by Ragnar Frisch and Frederick V. Waugh (1933) and later generalized by Michael C. Lovell (1963), the Frisch–Waugh–Lovell (FWL) theorem states that for any regression expression of the form

any coefficient 𝛃ⱼ can be obtained by:

1. Regressing Xⱼ on the set of other independent variables
2. Regressing y on the residuals from the preceding step, thereby yielding 𝛃ⱼ where

…and tilde Xⱼ denotes the residual obtained from regressing the independent variable Xⱼ on all the other independent variables in the model.

The key implication of the FWL Theorem is that the system of β estimates is separable — any subset of the βs can be solved by inferring within-group variation alone. Using the FWL Theorem, we can show algebraically that the β estimates obtained via demeaning are identical to the β estimates that we would normally obtain from the full model. We now have a powerful tool for reducing the size of the Gram matrix. For every dummy variable / fixed effect we can just subtract the means for each level from the response variable and any other regressors, dispensing with the dummy variable entirely.

This approach is not without drawbacks. Inverting the Gram matrix produces the joint covariance matrix which includes standard errors for every group intercept — omitting the fixed effects mean we no longer get these standard errors off the shelf. Also, we no longer get the point estimates from the fixed effects (albeit they are retrievable via a cheap post-estimation step). There is also the problem of how to approach multiple fixed effects. It turns out that there is a relatively straightforward solution to this, the method of alternating projections.

Method of Alternating Projections (MAP)

Through demeaning, we can control for the group characteristics that the fixed effect would otherwise need to capture. This leaves us with a much smaller and easier-to-fit multivariate model. Unfortunately, this process breaks down in the context of multiple fixed effects as the following example makes clear.

Say we are interested in modeling the number of batches shoppers deliver per week. Our sample data consists of three shoppers: shopper A, who tends to work in Smallsville, shopper B who accepts batches in Smallsville and Metro City, and shopper C who has only ever works in Metro City.

We want to fit two fixed effects — one at the shopper level, and another at the region level. Applying our earlier strategy of demeaning on the region, we get:

We create a new column with the number of batches-per-shopper after demeaning by region.

We know that our demeaning by region worked because if we take the mean of the demeaned number of weekly batches for each region (i.e. the rightmost column)…

…the two means are zero. We’ll repeat this process for shopper-level demeaning — starting with calculating the mean de-meaned weekly batches for each shopper (just taking the shopper-level means of the rightmost column above).

Applying this second stage of demeaning yields the values in the rightmost column below.

Here a problem emerges. If we take the region-level means again (i.e. the means of the rightmost column above)…

…we see that the regions no longer have a mean of zero — demeaning by shopper, effectively re-introduced the region-level information we wanted to remove. While this is problematic, let’s repeat this cycle of demeaning and see where it leads us. In the table below, we consolidate the demeaning that we’ve already done under the column “Iteration 1.”

Continuing the pattern, we subtract the updated region-level means — the results are in the column — “Iteration 2 (Demeaned on Region).” We’ll then re-update the shopper-level means as follows…

…and demean once again (see the rightmost column below).

Updating the region-level means, we get:

So the region-level means are still not equal to zero — we still haven’t removed the region-level information. However, notice how the distance from zero has halved relative to the initial iteration. Look what happens if we continue the pattern.

We can see that the distance between the demeaned values and zero halves with each iteration. More generally, the residual shrinks at a geometric rate — in this case, halving with each full iteration — until it falls below a tolerance threshold.

This is the method of alternating projections (MAP) first identified by Von Neumann (1950), who proved that alternating between two demeaning operations will always converge to the correct answer. Smyth (1996) refers to this approach as a full Gauss-Seidel partitioned algorithm, a.k.a. the “Zig Zag” based on the path one takes in traversing the subspace as shown in Figure 2. It was Guimarães & Portugal (2010) who flagged the applications of MAP for the estimation of high-dimensional fixed effects.

The key implication of MAP is that while demeaning across multiple sets of groups will invariably “undo” some of the orthogonalization achieved with the previous round of demeaning, we can simply add additional iterations to reduce the residual information to an arbitrarily small number. An interesting characteristic of the Zig Zag algorithm is that its speed depends on the structure of fixed effects. Hypothetically, if the fixed effects are orthogonal (for example if every shopper works in every region for exactly one time period), then the demeaning of one vs the other fixed effects leads to no contamination, and MAP should converge in one iteration.

As a final note, the actual implementation of this demeaning can be quite specialized. Fixest uses five arguments to parameterize its demeaning. Meanwhile, pyfixest offers seven distinct demeaning backends including Numba, Cupy, JAX, and Rust.

Applications

When taken together, the FWL Theorem and the Method of Alternating Projections provide a path for estimating high-cardinality fixed effects models with a fraction of the compute that would normally be required. We can see this clearly if we compare pyfixest to other estimators available from statsmodels and scikit-learn.

For our first point of comparison, we’ll look at statsmodels. For our baseline we will manually implement a single fixed effect by using dummy variables, increasing the cardinality from 20 to 20k categories. We use the Moore-Penrose pseudoinverse (pinv) estimation option as it is the default (although we recommend using QR-based estimation as pinv can hide duplicated variables and may not offer the best trade-off between performance and numeric stability). For our second trial, we’ll use the proper statsmodel API and invoke the dedicated mixedlm class. For our third trial, we’ll use linear regression via scikit-learn, once again creating dummy variables to implement the fixed effect. Our fourth and final trial is pyfixest.

One immediate takeaway is how quickly high cardinality fixed effects become computationally intractable given conventional methods. When we have cardinality in the tens of thousands, memory consumption quickly becomes untenable. Using statsmodel’s intended API keeps memory consumption at more reasonable levels (<100mb), but run time still scales polynomially with the cardinality of the fixed effect. Note how this is less of a problem for scikit-learn, as it stores the dummy variables within a sparse matrix. When processed by the Least Squares with QR factorization solver (LSQR) scikit-learn uses memory at the rate of O(N) (as opposed to O(N·k)).

We can also see that while the dummy variable implementation in scikit-learn may match pyfixest’s performance initially, scikit-learn’s sparse solver will continue to fall behind pyfixest as cardinality increases. Most importantly, our method of deploying dummy variables via scikit-learn will not scale to interactions across multiple fixed effects — something that pyfixest supports natively.

What are the practical implications of leveraging methods that enable high-cardinality fixed effects? As an example, we looked at a recent experiment run by Marketplace where Instacart removed a specific type of delivery window option. For this static-geo test, we wanted to know whether there was any change in the average amount of time it took to fulfill an order (we refer to this as ‘active time’). Figure 4 shows the relative sizes of the standard errors for our mainstay difference-in-differences two-way fixed effects model (generated by Pyfixest) versus simple bivariate regression.

Thanks to our ability to deploy high-cardinality fixed effects, we get a 4.1x reduction in the metric’s relative standard error — the equivalent to having ~17x more observations in the bivariate regression model. It would be difficult to envision running static-geo tests without leveraging fixed effects.

Conclusion

For Marketplace experimentation, the stakes are high when it comes to flagging statistically defensible movement in key metrics. The aforementioned 4.1x reduction in standard error size means the difference between an experiment that can detect a meaningful effect within a reasonable timeframe and one that cannot. Static-geo tests, which are already operating at a fraction of the sample size of a standard A/B test, are particularly sensitive to this.

Without high-cardinality fixed effects, many of the experiments we run today would simply be underpowered. At the same time, deploying the necessary fixed effects for a large number of levels quickly becomes problematic for software packages not explicitly designed for this use case. By leveraging the FWL Theorem addition to the method of alternating projections, packages like Fixest and Pyfixest enable the modeling of high-cardinality spaces, allowing Instacart to get the most accurate insight possible.

References

• Bergé, L. R., Butts, K., & McDermott, G. (2026). Fast and user-friendly econometrics estimations: The R package fixest. arXiv. https://doi.org/10.48550/arXiv.2601.21749
• Frisch, R., & Waugh, F. V. (1933). Partial Time Regressions as Compared with Individual Trends. Econometrica, 1(4), 387. https://doi.org/10.2307/1907330
• Gaure, S. (2013). OLS with multiple high dimensional category variables. Computational Statistics &amp; Data Analysis, 66, 8–18. https://doi.org/10.1016/j.csda.2013.03.024
• Guimarães, P., & Portugal, P. (2010). A simple feasible procedure to fit models with high-dimensional fixed effects. The Stata Journal, 10(4), 628–649. https://doi.org/10.1177/1536867X1101000406
• Lovell, M. C. (1963). Seasonal Adjustment of Economic Time Series and Multiple Regression Analysis. Journal of the American Statistical Association, 58(304), 993–1010. https://doi.org/10.1080/01621459.1963.10480682
• The PyFixest Authors. (2025). pyfixest: Fast high-dimensional fixed effect estimation in Python [Computer software]. GitHub. https://github.com/py-econometrics/pyfixest
• Smyth, G. K. (1996). Partitioned algorithms for maximum likelihood and other non-linear estimation. Statistics and Computing, 6(3), 201–216. https://doi.org/10.1007/BF00140865
• Von Neumann, J. (1950). Functional Operators (AM-22), Volume 2: The Geometry of Orthogonal Spaces. (AM-22). Princeton University Press. http://www.jstor.org/stable/j.ctt1bc543b

Leveraging PyFixest for High-Cardinality Marketplace Modeling at Instacart was originally published in tech-at-instacart on Medium, where people are continuing the conversation by highlighting and responding to this story.

Key Contributors: Karuna Ahuja, Marko Avdalovic, Soroush Sobhkhiz, Shrikar Archak, Xiyu Wang, Ji Chao Zhang, Hao Yan

Introduction

Every time a user opens Instacart, they see product recommendations: on the retailer home page, in search results, and alongside their cart. Many of these recommendations are sponsored products surfaced by a retrieval model that decides which products to show from a vast ads product catalog. A relevant ad helps users discover products they didn’t know they needed; a less relevant one generates friction.

Two years ago, we introduced Contextual Recommendations (CR), a BERT-based sequence model powering retrieval for both ads and organic recommendations across all major browse surfaces. In this post, we’ll focus on our ads retrieval. We will detail how we rebuilt the system, by moving from an encoder that scores products to a generative model that spells them out, token by token. By doing so, we unlocked a new level of contextual matching — ensuring brands appear exactly when users want them, while simultaneously opening up discovery of thousands of relevant products the previous system couldn’t retrieve.

Contextual Recommendations: A recap

At its core, CR treats grocery shopping as a language modeling task, where atomic product IDs function as tokens and, the finite subset of the catalog it is trained on, acts as its ‘vocabulary’.

The model leverages the user’s real-time session, which includes product views, item page visits, and cart additions, as a sequence of these product tokens. A BERT-like transformer is then trained on millions of authentic shopping sessions to predict the next token (i.e. singular product) in the sequence. This process allows the model to learn and capture complex purchasing patterns, such as the tendency for users who add pasta and olive oil to frequently add garlic next.

This single retrieval layer replaced multiple ad-hoc systems and powers recommendation carousels across all major browse surfaces, serving both ads and organic content. At inference time, it scores every product ID in its vocabulary against the current session and returns the top K products.

For the full technical details, see our previous blog post.

When Scoring Stops Scaling

Since launching CR, we iterated on two fronts to improve the underlying model; improving the coverage of our catalog and adding more context.

First, we expanded the product vocabulary the model trains on. This helped us expand the retrieval coverage. Second, we added richer context through retailer awareness and long-term user personalization. Both the upgrades led to meaningful gains in add-to-carts and ads coverage, particularly for specialty retailers and short shopping sessions.

Each of the above improvements operated within the same fundamental architecture: score every product in a candidate set, return the top K products. However, as the catalog grew and user shopping journeys became more diverse, this architecture presented three constraints that placed a ceiling on our discovery potential, especially for our ad recommendations:

The vocabulary bottleneck: The CR model relies on atomic product IDs as distinct tokens, which establishes the boundaries of what the model can interpret and predict. While expanding this vocabulary enhances the model’s ability to understand the detailed context of a user’s session, it simultaneously increases model size and latency while creating data sparsity for less common items. Additionally this catalog is non-stationary. As new products are added to the catalog, the coverage gap keeps expanding. Consequently, relying solely on vocabulary expansion proved insufficient for representing the full breadth of the catalog, as specialized products often remained outside the model’s recognizable token set.

The ‘cold start’ hurdle: To train this model, the historical shopping sessions were designed as sequences of atomic product IDs. This occasionally caused it to memorize co-occurrences instead of learning generalized associations based on the user’s intent. This resulted in the model favoring high-frequency items over newer products which are more aligned with the user’s context. For instance, while a user is building a cart toward a summer barbecue [eg: ground beef, hamburger buns, lettuce], the previous system had a tendency to default to a generic grocery staple [eg: milk] rather than surfacing an emerging brand’s condiment [eg: mustard] that fits the intent better. This collaborative filtering approach, while effective at a baseline, often lacked the responsiveness of the model to recommend products based on what the user is actually doing right now.

The structural drift: The final candidate set from the model is generated by predicting a probability distribution across the entire vocabulary of product IDs. Without a built-in hierarchy to keep the recommendations focused, the model occasionally retrieves a disjointed mix of items. For example, a breakfast-themed cart [e.g., milk, eggs, cereal] may lead to laundry detergent being retrieved along with other valid recommendations [e.g., bread, muffins]. If the subsequent ranking model was miscalibrated on these outlier products, these incoherent recommendations from the candidate set would eventually get bubbled up to the user next to a perfectly good set of recommendations.

These technical constraints ultimately limited how well we could connect users with the full breadth of our ads catalog, resulting in missing tail categories, narrower brand representation, and the occasional misaligned recommendation. We needed to rethink the product representation, the architecture, and the retrieval mechanism, not just add more features to the same model.

Teaching the Model to Spell

Our new approach is inspired by TIGER (Google DeepMind), a method that demonstrates a model’s ability to generate the semantic tokens of the next relevant item, rather than merely scoring a predetermined set of candidates. This generative paradigm has been adopted in production by companies such as Spotify (GLIDE, NEO) and YouTube (PLUM).

However, Instacart’s ad retrieval presents distinct challenges rooted in the unique nature of grocery shopping. Unlike platforms where the user’s intent is narrow, Instacart users often manage a highly diverse shopping list including items from fresh food to cleaning supplies and pet care — sometimes all within a single session. The user’s intent shifts mid-cart. Users shop across various retailers on our marketplace, each with a unique product catalog.

To address this, our model must look beyond historical purchases; it must also account for the real-time dynamics of the active shopping session. This is also where we have an opportunity to unlock new product discovery. Instead of just picking some atomic product IDs from a massive list, we needed a model that could look at the user’s cart, leverage its learned semantic concepts, and “autocomplete” the rest of the session. By generating an abstract concept rather than a specific item, the model shifts from memorization to generalization. This instantly connects the shopper’s intent to the full depth of our catalog, even for products with zero transaction history.

Building this for grocery required two things: a new product vocabulary, and a new way to use it.

Instacart Semantic IDs: A new Product Vocabulary

Before we could build this new retrieval system, we needed to change how we represented products. This motivated us to invest in building Instacart Semantic IDs.

Instacart Semantic IDs, SIDs, replace atomic product IDs with short sequences of codewords generated by an RQ-VAE. A product’s SID looks like 35_7_120_184: four tokens from learned codebooks at different granularity levels. Semantically similar products share prefixes:

35_7_119_493 → Organic Good Seed Thin Sliced
35_7_120_184 → Artisanal Italian Bread
35_7_120_185 → Classic Italian Bread

This essentially compresses the product vocabulary, as multiple very similar products are represented by a single SID. Leveraging this new representation in our retrieval model provided three major benefits:

• SIDs provide coverage to every item in the catalog, regardless of whether it has a historical purchase history. A new product entering the catalog is added to one of the existing SIDs and is visible to the model from day one.
• The model learns to generalize sequences better based on semantic codewords instead of simply learning specific product co-occurrences.
• The embedding parameter space within the model is decreased by 125x.

The Context Template: A new Training Corpus

This new compact SID format also fundamentally changes how we construct our training data.

In the previous model, mapping atomic product IDs consumed the entire token capacity. By shifting to SIDs, we freed up the massive dictionary space, allowing us to design a richer input training corpus based on Instacart domain data. We achieved this by mining millions of historical shopping sessions and enriching them with new context tokens. We formulate the session into the following sequence template:

Each segment of this prompt serves a distinct role, and they are separated by special tokens:

• A retailer type token tells the model which catalog and shopping context the user is shopping in. Because our marketplace retailers span grocery, pet, beauty, home goods, and more, this token helps us capture the distinction.
• User history SIDs from past purchases capture long-term preferences. By taking the top N previously purchased SIDs and expressing them in the same token format the model generates in, we seamlessly connect past behavior to future predictions.
• Cart SIDs capture the real-time intent of the current session. While user history tells the model what someone typically likes, the cart SIDs tells it what they are building today, adapting as new items are added.

During training, the model reads this template and learns to autoregressively generate the SID of the next item the user adds to their cart. The template structure also gives us a clean interface for future signals (such as occasion awareness, search queries, page type) without architectural changes. Each new signal is simply a new segment in the prompt.

From Input to Candidates: A new Retrieval Paradigm

During serving, we build the candidate set via beam search. As illustrated in the diagram below, the decoder reads this input and generates recommendations token by token. At each step, beam search explores multiple promising paths for the next codeword. This ultimately yields several distinct, fully formed SID sequences. Finally, these generated sequences are mapped against a retailer-partitioned index to retrieve a diverse variety of relevant, available ad products.

This new paradigm, combined with the catalog coverage SIDs already provide, directly addresses the three limitations we hit with the previous solution:

• Eliminating the bottleneck: By generating sequences from a small, fixed set of codewords rather than scoring an ever-expanding list of product IDs, the scaling constraints of the vocabulary bottleneck disappear. The model constructs the semantic representation of the next item on the fly, avoiding the memory and latency penalties that previously restricted our catalog coverage.
• Inherent structural coherence: Generating auto regressively means each codeword is explicitly conditioned on the previous one. This enforces a strict hierarchy during retrieval. If the model begins generating a prefix for “Produce,” the beam search remains confined to that semantic neighborhood, actively preventing the random outlier leakage caused by flat probability distributions.
• Dynamic diversity dials: Unlike scoring models, the generative approach unlocks direct tuning mechanisms through beam width and temperature sampling. These serve as precise levers to balance intent and exploration — allowing us to dial up strict precision on search pages, while turning up brand diversity and discovery on post-checkout surfaces.

Rebuilding Serving Infrastructure

As autoregressive decoding with beam search is fairly compute intensive, it was not viable to serve this model the legacy serving stack that relied on Python and CPU inference. To unblock this model serving, the team developed a brand new GPU serving stack. This new system leverages TensorRT-LLM for high-performance inference and is deployed on Nvidia’s Triton Inference Server.

The new serving stack represents a fundamental shift in architecture. Implemented as a Go-native service, it delivers higher throughput and lower latency compared to the legacy Python environment. It is fully integrated with Griffin 2.0, Instacart’s machine learning serving platform, streamlining deployment and maintenance within our ecosystem.

How the New Stack Works

This new serving stack performs a series of critical, high-speed operations to power the new ads retrieval system:

• Input Translation: Features are dynamically fetched and collated to create the input prompt.
• GPU Model Inference: The model runs inference and generates relevant SID sequences.
• Product Mapping and Indexing: Finally, the generated SIDs are mapped back to active ad products via a specialized, highly efficient retailer-partitioned index, ensuring that only relevant, available, and correctly attributed ads are retrieved.

Despite producing approximately 2x the candidate volume, mean retrieval latency decreased by 10–17%, validating the investment in the new GPU serving architecture.

Measuring the Impact

We launched this system to power ads carousels on two discovery surfaces that bookend a user’s shopping journey: a retailer home page at the very beginning, and the pre-checkout phase just before the order is finalized. These are contexts where users are browsing rather than searching, and candidate diversity & contextual relevance matter more than surgical precision.

In our online A/B tests against the incumbent model, the generative approach delivered a +5% improvement in click-through rate. What was truly transformative, however, was the +34% step-function increase in add-to-carts. This indicates that we didn’t just capture more clicks; we significantly elevated the quality of that engagement, converting passive exploration into tangible downstream conversions.

While the quantitative metrics showcase the remarkable impact of this new solution, the qualitative improvements are just as striking. Here are a few ways we saw our recommendations evolve.

What the New Recommendations Look Like

Tail Category Alignment

The generative architecture allows the model to generalize more effectively than its predecessor, resulting in deeper session understanding and recommendations that align more closely with a user’s active shopping trip.This improved performance is most evident in “tail” segments such as beauty and pet care. Previously, hard vocabulary constraints often filtered out these specialized items, causing the system to fallback on high-frequency, generic grocery staples.

For example, a customer purchasing pet food at a big box retailer now receives pet-specific recommendations instead of broader grocery suggestions. This hyper-relevant, session-aware targeting is a primary driver behind the engagement movement, as users are finally seeing ads that actually match the current intent of their cart. Here’s a sample anonymized example from real production logs.

Improved Brand Diversity: The Most Impactful Outcome

Building on this enhanced session understanding, the new architecture also unlocked a substantial boost in brand diversity by reaching deeper into our catalog. By overcoming the limitations of a fixed token space, TIGER recommended 2.7x more brands and 1.8x more sub-categories than the previous system. This demonstrates that the model goes beyond simply relying on the transaction history of well-known SKUs. Instead, it dynamically identifies and suggests emerging brands that align with a user’s specific intent, significantly enhancing product exploration.

We saw the most substantial diversity gains in highly dense categories, driving improvements of +421% in Alcohol, +396% in Beverages, and +229% in Healthcare. In these categories, the previous solution’s architectural ceiling prevented these products from being retrieved.

This unlocks new potential for Instacart’s ads ecosystem, creating a valuable opportunity for emerging brands to drive growth by surfacing their products in highly contextual placements.

What’s Next

With Instacart Semantic IDs as the vocabulary and this new generation engine, we’ve successfully demonstrated the move from scoring a fixed candidate set to producing recommendations autoregressively from user context. Here’s where it goes from here.

SID quality: The quality of the codebook is fundamental to everything downstream, impacting retrieval precision, brand diversity, and coherence. Future improvements include multi-resolution codebooks, co-occurrence contrastive regularization, and incorporating dietary constraints into the initial codebook level. A full design space is covered in our companion post [SID].

Richer context engineering: Now that our template can seamlessly absorb new signals, we are focusing on feeding the model higher-order intent. By injecting real-time search queries, specific page contexts, and detected shopping occasions directly into the prompt, we can push the model to generate even more surgical, intent-driven recommendations.

From retrieval model to discovery platform: This initiative has given us a generative model operating over product tokens. The natural next step, and one that YouTube’s PLUM has validated at scale, is a multilingual model that reasons across both SID tokens and natural language. For us, this means learning from diverse grocery domain data like search queries, product descriptions, and shopping occasions alongside cart sequences. A model that can reason across both vocabularies opens up instruction-following interfaces, set-completion training, and richer user modeling. And ActionPiece (Google DeepMind) has shown that user actions, not just items, can be tokenized and generated in a context-aware manner, hinting at a future where the same architecture could power what we show users next: a reorder nudge, a recipe suggestion, or a discovery carousel?

The core architecture doesn’t change; the training recipe does. We are no longer just retrieving ads — we are building an AI that fluidly speaks the language of grocery, creating a richer, more intuitive discovery experience for users and advertisers alike.

We’re early in this journey, but the foundation is in place.

Acknowledgements and Final Notes

We would like to extend deep gratitude to our cross-functional partners Chang Zhang, Walter Tuholski, Trevor Yao, Cheng Jia, Joseph Haraldson, Nick Cooley, Tristan Fletcher who have provided critical ongoing design feedback and driven system integrations to bring this research to production.

References

• Semantic IDs: Product Understanding at Scale: Instacart Tech blog
• Sequence Models for Contextual Recommendations at Instacart: Instacart Tech blog
• TIGER: Generative Retrieval for Recommendations (Google DeepMind)
• PLUM: Pre-trained Language Models for Industrial-scale Generative Recommendations (YouTube)
• ActionPiece: Contextually Tokenizing Action Sequences for Generative Recommendation

From Scoring to Spelling: Rebuilding Ads Retrieval at Instacart was originally published in tech-at-instacart on Medium, where people are continuing the conversation by highlighting and responding to this story.

Key Contributors: Shrikar Archak, Karuna Ahuja, Soroush Sobhkhiz, Marko Avdalovic, Xiyu Wang, JiChao Zhang, Hao Yan, Chris Hartley

Introduction

Operating a grocery catalog at Instacart’s scale means managing millions of products across thousands of categories. Every product is assigned to a category in our hierarchical taxonomy like “Dairy > Cheese > Parmesan”. These categories provide broad classification, but they miss the connections that drive how customers actually shop.

For example, a customer is building a cheese board. They’ve added Parmigiano Reggiano, and now they need accompaniments. Our taxonomy puts it in “Dairy > Cheese > Parmesan,” so a category-based system can suggest other parmesan cheeses. But it can’t connect them to the Castelvetrano olives in Pantry > Condiments > Olives, the olive tapenade in Deli > Olives Dips and Spreads, or the crudité and pre-assembled cheese tray in Deli > Prepared Meals > Party Trays. These products live in completely different branches of the catalog, with no shared ancestor below “Food.” But any customer would tell you they belong together.

This cross-category blindness shows up in three ways.

Cold start: new products arrive with zero purchase history. We can assign them to the right category, but a category alone can’t connect them to the products customers would actually consider alongside them, so they stay invisible.

Tail category coverage: recommendation models learn from volume, so they skew toward popular grocery staples. Products in sparse categories lack the interaction data to surface, and the taxonomy gives the model no bridge to related items in other branches.

Catalog quality at scale: with millions of products, mislabeling is inevitable — a protein bar filed under “Candy,” a sparkling water under “Soda.” A rigid tree has no way to flag these because the only signal is the label itself.

In this post, we walk through how we built semantic IDs at Instacart to address these problems: the embedding choices, the contrastive training approach that leverages our catalog structure, the two-flavor strategy for precision vs. discovery, and what we learned when things didn’t work.

What a Semantic ID Looks Like

A semantic ID is a short sequence of integers generated by compressing a product’s embedding through a residual vector quantizer. Products with similar meaning share prefixes; products that differ split at progressively finer levels.

Source: Recommender Systems with Generative Retrieval (TIGER)

Here’s what this looks like in practice. Under semantic ID prefix 6_19, our system groups:

6_19_32 → Italian cheeses (Parmigiano, Pecorino, Mozzarella, Ricotta)
6_19_24 → Specialty cheeses (Brie, Manchego, Halloumi, Goat cheese)
6_19_12 → Olives (Castelvetrano, Kalamata, olive medleys)
6_19_7  → Tapenades (olive tapenade, spreads)
6_19_9  → Deli trays and dips (crudité trays, cheese dips)
6_19_14 → Croutons

No one wrote a rule connecting Pecorino Romano to Kalamata olives to olive tapenade. The model learned that these products inhabit the same culinary universe, spanning Dairy, Pantry, and Deli departments, by compressing their embeddings into codes that share a prefix.

Zooming into one branch shows how the hierarchy captures finer distinctions:

6_19_32_4  → Fresh Mozzarella, Mozzarella Bars
6_19_32_16 → Crumbled Gorgonzola, Blue Cheese Crumbles
6_19_32_63 → Hard Italian cheeses (Parmigiano, Pecorino, Asiago)
6_19_32_70 → Ricotta Salata

Same first three levels: these are all Italian cheeses. The fourth level captures the functional distinction between fresh, crumbled, hard aged, and ricotta. A customer out of Pecorino Romano might accept Parmigiano Reggiano (same L4, group 63) before reaching for Gorgonzola crumbles (different L4, 16, but same L3).

The Research Landscape

The idea of discrete learned codes for retrieval isn’t new. Google DeepMind’s TIGER introduced semantic IDs for generative recommendation. YouTube’s PLUM extended this to production scale with behavior-aligned codebooks. Mender explored mixed semantic enhancement, and BBQRec showed how multi-modal signals can inform quantization.

Grocery is a different domain: users fill multi-category shopping lists in a single session, the catalog spans millions of products, and the taxonomy structure we already maintain gives us a supervision signal that media recommendation systems typically don’t have.

From Embeddings to Codes

Semantic IDs are built on top of product embeddings, high-dimensional vectors where similar products end up nearby in vector space. (For more on our embeddings, see [How Instacart Uses Embeddings to Improve Search Relevance].)

With millions of products, raw embeddings present practical challenges: significant memory, expensive nearest-neighbor search, and incompatibility with discrete token-based systems like LLMs. We compress them using a Residual Vector Quantizer (RQ-VAE), which learns a hierarchical codebook by iteratively quantizing residuals. Each level captures finer distinctions than the last. Our 4-level setup produces IDs like 6_19_32_63, where Level 1 might separate beverages from cleaning supplies and Level 4 distinguishes between brands of hard Italian cheese.

But compression alone isn’t enough. A vanilla RQ-VAE optimizes for reconstruction fidelity. It has no notion of product relationships.

Teaching the Quantizer What “Similar” Means

Without structural guidance, the quantizer produces two problems: fragmentation (two marinara sauces that any customer would consider substitutes end up in different branches) and error propagation (a product with product details, category and descriptions gets embedded poorly and placed among irrelevant items). These embeddings are generated from a product’s text, its name, brand, attributes, size description, and category path — using our in-house ESCI (Exact, Substitute, Complementary, Irrelevant) model, which learns representations from search relevance data.

Contrastive Regularization with Catalog Structure

Inspired by PLUM’s behavioral alignment approach, we added a contrastive term to RQ-VAE training, using our catalog taxonomy as the supervision signal rather than engagement data (which isn’t available for cold-start products). A contrastive loss works by pulling similar items closer in the learned code space and pushing dissimilar items apart.

Rather than binary same/different labels, we define relatedness along a gradient based on where two products sit in the catalog tree. Two products in the same leaf category (say, two marinara sauces from different brands) are strong positives because they share the most specific category. Products in sibling categories (marinara and alfredo, both under “Pasta & Pizza Sauces”) are moderate positives because they share a parent but not a leaf. Products with no shared ancestor (“Pasta Sauce” vs “Office Supplies”) are negatives. The signal isn’t relative to any single product; it’s defined by the structural distance between any pair in the taxonomy.

Hierarchical Batch Sampling

The contrastive loss only works if each training batch contains both related and unrelated products. With random sampling over millions of items, most batches would be entirely unrelated — the loss would have no positive signal to learn from.

We fix this by constructing batches deliberately. First, we pick a random parent category (say, “Pasta & Pizza Sauces”). We fill roughly half the batch with products from its child categories — marinara, alfredo, pesto — so the batch naturally contains sibling pairs. We fill the other half with products from unrelated categories (laundry detergent, dog food) to provide hard negatives. Within each category slot, we sample multiple products, so same-leaf pairs (two marinara sauces from different brands) appear automatically. No explicit pair labeling is needed — the catalog structure does the work.

The Loss

L_total = L_reconstruction + L_rq + λ · L_contrastive

The loss aligns embedding similarity with codebook index similarity across all four levels. Coarser levels (L1, L2) are weighted more heavily so broad groupings take priority. With λ = 0.01, the contrastive term is a gentle regularizer: strong enough to improve coherence, weak enough not to destabilize reconstruction.

What’s next: incorporating engagement-based signals (substitution patterns, co-purchase data) following PLUM’s approach.

Two Flavors: Precision vs. Discovery

The two approaches differ starting at the input.

ESCI (precision) embeds raw product text (name, brand, description, size, some attributes and categories) through our search relevance model, which was trained on query-product matching. The result: embeddings tuned for “is this the same thing the customer asked for?” This produces tight clusters where every item is a direct substitute, like Whole Bean Coffee (0_8_55_72), where each product is a medium roast from a different brand, interchangeable for any customer who wants whole bean coffee. ESCI powers substitution, search, and reordering.

ESCI+Gemma (discovery) takes a different path. It first runs the product through Gemini Flash (~10x faster, ~5x cheaper than full-size models) to extract structured attributes (product type, key ingredients, dietary tags, format), stripping away marketing copy along with the metadata used for ESCI. It then embeds that cleaned representation with Gemma, an off-the-shelf embedding model. The goal is to test whether a general-purpose model, given cleaner inputs, can capture nuances that a domain-specific model misses. The result: broader clusters that capture lifestyle and usage patterns. ESCI+Gemma powers homepage feeds, cross-selling, and exploration.

Neither is universally better. The key is matching the right flavor to the right surface.

How We Know It Works

We measure semantic ID quality directly rather than relying solely on downstream metrics.

Similarity-depth correlation. We measure the relationship between embedding similarity and shared semantic ID levels. Correlations of 0.69–0.84 confirm the semantic id hierarchy captures meaningful structure. Among highly similar pairs (≥0.9 cosine similarity), 98–99% share Level 1, declining to 18–37% at Level 4. This is expected, since Level 4 distinguishes between very similar products.

LLM-based cluster evaluation. Quantitative metrics tell us whether the hierarchy is structurally sound, but not whether the clusters make functional sense. To assess that, we prompt LLMs to look at each leaf group and score it on three dimensions: functional coherence (do these products serve similar purposes?), purchase likelihood (would a customer buy these together?), and customer journey relevance (do they fit the same shopping context?). This gives us a scalable proxy for human judgment across thousands of clusters. ESCI scores higher on substitutability; ESCI+Gemma excels at thematic coherence, matching their intended use cases.

Taxonomy alignment. We check whether products sharing a Level 1 code also share a top-level category. Most do, and the misalignments turned out to be more valuable than the alignments.

Where It Breaks, and What That Tells Us

When sparse text produces divergent codes. Two Riesling wines (0_19_52_63 and 0_31_52_88) share the same category path and 0.86 cosine similarity, yet diverge at L2 due to sparse descriptions. A team branded t-shirt (1_19_21_20) and generic team apparel (1_7_41_59) had 0.95 similarity but matched only at L1. One has a detailed description, the other just four words.

The pattern: sparse or inconsistent text leads to degraded embeddings, which lead to divergent codes. Products with rich descriptions and complete catalog metadata produce more stable codes. Products with minimal text give the embedding model little to work with, and the quantizer faithfully compresses that noise into divergent codes. Enriching product data for these sparse items is an ongoing effort.

When the system is right and the category is wrong.

Sometimes a product’s semantic ID disagrees with its taxonomy label. A “Protein Bar” labeled under “Candy” clusters with other protein bars in “Sports Nutrition.” A “Sparkling Water” filed under “Soda” lands among other sparkling waters. In each case, the semantic ID placed the product where it functionally belongs. The error was in the taxonomy, not the code.

This turns semantic IDs into an automated catalog audit. Any product whose cluster assignment disagrees with its category label is a candidate for correction. We’re building this into a pipeline: automated flagging of code-vs-label mismatches, confidence scoring for how strongly a product fits its cluster versus its label, and prioritized review queues for human verification. What started as a recommendation primitive is becoming infrastructure for ongoing catalog health.

What Semantic IDs Unlock

Building this system taught us a few things that generalize beyond our specific implementation:

The embedding is the decision. The RQ-VAE compresses whatever structure the embedding space gives it. Choose your embedding based on the business problem.

Catalog structure is a free supervision signal. PLUM showed behavioral signals are powerful; we showed catalog structure gets you surprisingly far even before you have behavioral data.

Standardize before you embed. Lightweight attribute extraction is a high-ROI preprocessing step that reduces noise throughout the pipeline.

Evaluate codes directly. Downstream metrics can mask systematic quality problems. Intrinsic evaluation catches issues before they compound.

From Vocabulary to Language

With ~2,000 codeword tokens representing the entire catalog, generative retrieval becomes possible: a model that produces the semantic ID of the next relevant product, codeword by codeword, conditioned on the user’s context.

We first proved this on product carousels, where the generative approach delivered +34% add-to-carts and surfaced products from 2.7x more emerging brands. Tail categories saw the largest gains, precisely because semantic IDs gave those products a representation the old model couldn’t.

Those carousel results were the starting point. Semantic IDs now power product retrieval, replacement recommendations, and next-item prediction across Instacart. Looking ahead, we’re bringing them to product detail page recommendations, cart assistant suggestions, and ranking features, particularly to address cold start where they have the most leverage.

The broader lesson: semantic IDs started as a compression technique for making embeddings compatible with discrete systems. They became something more, a shared vocabulary that lets every model in our stack reason about product relationships in the same language. The more surfaces that speak this vocabulary, the more value each one gets from it.

Acknowledgements and Final Notes

We would like to extend deep gratitude to our cross-functional partners Trace Levinson, Pradeep Karaturi, Shishir Prasad, Tristan Fletcher, Vinesh Gudla, Raochuan Fan, Xiao Xiao, Prakash Putta who have provided critical ongoing design feedback and driven system integrations to bring this research to production.

References

• TIGER: Generative Retrieval for Recommendations (Google DeepMind)
• PLUM: Pre-trained Language Models for Industrial-scale Generative Recommendations (YouTube)
• Mender: Generative Recommendation with Mixed Semantic Enhancement
• BBQRec: Behavior-Bind Quantization for Multi-Modal Sequential Recommendation
• How Instacart Uses Embeddings to Improve Search Relevance (tech.instacart.com)
• Eugene Yan: Semantic IDs

Semantic IDs: Product Understanding at Scale was originally published in tech-at-instacart on Medium, where people are continuing the conversation by highlighting and responding to this story.

Levi Boxell, Tilman Drerup, Alexandr Lenk

The Economics Team at Instacart is an applied science team that operates at the intersection of machine learning engineering and economics. Similar to other applied science teams, our work involves a good chunk of engineering, steeped in statistics, math, theory, and strategy. And while that is still at the heart of what we do today, the surprisingly rapid emergence of artificial intelligence has also fundamentally altered our work in ways that we did not see coming.

With this post, we want to provide a brief check-in and share an analysis of the patterns we are seeing from a distinctly economic perspective. To do so, we analyze the empirical dynamics of our project portfolio between 2023 and today, looking at the evolution of both the nature and quantity of our work over time. To start, let’s have a quick refresher of what economists at Instacart do and provide a theoretical framework to think about the impact of technological change through AI.

Background & Theoretical Framework

At Instacart, economists spend their day-to-day on a diverse portfolio of tasks and activities. Similar to other applied science teams within the company, our work relies on a blend of skills, including economics, statistics, math, machine learning, data manipulation, coding, and AI. Due to this versatility in tasks, the team’s work provides a particularly rich testing ground for predictions derived from economic theories concerning the impact of technological change.

But what does economic theory actually tell us? A useful theoretical abstraction for an applied scientist’s role is to frame it as a bundle of tasks (Autor, Levy, and Murnane, 2003), with each task characterized by its own production function (Acemoglu and Autor, 2011). Slightly simplified, a production function tells us how much output we can produce for a given level of input in a specific task. Comparisons of production functions across tasks in turn determine how we allocate our time and effort. Recent advances in AI have meaningfully affected the production functions associated with a wide variety of tasks, with bigger effects on some (e.g., coding) and currently smaller effects on others (e.g., causal inference). In consequence, we should see meaningful shifts in the distribution of tasks in our portfolio that reflect the differential effect AI has had on different types of tasks (Acemoglu and Restrepo, 2018 and 2019). Notably, in certain types of tasks, AI may even have discontinuous effects as it raises an applied scientist’s competence above thresholds of minimum viable competence, making entirely new task categories feasible.

To exemplify and make this a bit more concrete, AI is already delivering large productivity gains in many pattern-matchable, high-volume tasks, including the development of standard ML pipelines, data transformations, and boilerplate code. In such tasks, applied scientists should be able to produce more output (with less time). This should free up capacity that can be reallocated to other tasks. At the same time, entirely new tasks may enter an applied scientist’s portfolio as AI allows them to reach the minimal competence required to execute on such tasks. For example, an applied scientist without experience in writing frontend code might now be able to produce functional web applications. Similarly, a scientist with no experience in configuring CI/CD pipelines should now be able to stand them up.

Let’s jump into the data to see whether we can find evidence that is consistent with such predictions.

The Data: GitHub Contributions

Our analysis is based on three years of our team’s GitHub activity between 2023 and 2025. While we had some turnover during this time, our hiring criteria did not materially change, so any differences in output or focal areas documented below should not reflect team composition or selection effects. Importantly, we did not explicitly recruit specialists in any newly emerging task categories. In our analysis, we will focus on PRs and lines of code written per person as our primary output metrics. Evidently, these are far from perfect measures of productivity as they do not take quality of output into account. Having said that, we have a lot of anecdotal evidence of individuals being able to achieve milestones at a faster pace and with comparable quality. So while this covers some components of productivity, it’s far from exhaustive and there is plenty more to the job than what is captured in these patterns. At the same time, the distributional shifts we highlight below provide some evidence that what we see reflects true productivity adjustments in accordance with theoretical expectations.

To analyze our Github activity, we used an LLM to classify every pull request made by the team between 2023 and 2025. The classifier assigned each PR to one of eight categories (ML/Model Dev, Data Pipelines, Platform/Tooling, Analysis/Research, Infra/DevOps, Frontend/UI, Experimentation, and Other) based on the PR’s title, description, and changed files. All productivity metrics are reported relative to the 2023H1 baseline.

Productivity

Let’s first look at overall productivity in terms of code contributions, one of the most visible signs of an applied scientist’s raw output. As is evident from the two plots above, we saw a massive increase in the number of PRs and LoCs, each essentially doubling relative to their 2023H1 respective baseline. This overall tendency is strongly consistent with a meaningful upward shift in the team’s general productivity. Notably, the effects seem to become larger over time, likely reflecting the release of increasingly powerful tooling. We see a first small increase in the second half of 2023, following the release of new AI-powered productivity tools like Ava and their growing popularity within the team. A much more pronounced jump shows up in the first half of 2025, coinciding with the team’s broader adoption of Cursor agents. In 2026 (not shown), we saw a further substantial acceleration as we began to integrate Claude into our workflows, with the number of monthly Claude usage days across the team increasing by more than 400% (!) between January 2026 and April 2026 alone.

Differential Skills Drive Redistribution and Task Diversification

When we look at the distribution of tasks performed by individuals on the team, one thing immediately stands out: The average number of unique categories of tasks per member increased by 33%, adding approximately 1.3 new categories per member. At the same time, the average share of work outside each person’s primary task category (the one they focus most of their effort on) rose by ~37%. These results suggest that the team has started to work on a more diverse and less concentrated set of tasks.

To better understand what’s happening here, the table below presents a breakdown of the distribution of specific tasks in the team’s portfolio and how it changed over time. What’s behind these changes and does economic theory provide an explanation?

As we mentioned above, we expect AI to have a highly heterogeneous effect on the production functions associated with different types of skills, raising productivity in particular in areas that are more boilerplate and pattern-matchable. If that is indeed the case, the share of time allocated to such tasks should drop, freeing up capacity to work on alternative tasks. This is exactly what we find: More standardized tasks under Core ML (Data Pipelines or ML/Model Dev) saw declining shares, falling from 73% to 64% of the team’s PRs. Similarly, we observe lower shares for traditional productionization infrastructure tasks, dropping from 7.8% to 4.1%. Importantly, these share decreases coincide with an increase in overall output associated with such tasks. In essence, it seems that more standardized tasks now require substantially less time, allowing the team to do more of them while also freeing up capacity to work on other tasks.

And where did that freed capacity go? One particularly interesting theoretical prediction is that AI raises competence above what is minimally viable on tasks previously outside a worker’s skill set. Applied to our context, this means that scientists can now take on tasks outside of the core scientist skill set as the required specialized knowledge is becoming more easily accessible. And this is exactly what we see when we look at the categories that rise to prominence in our portfolio: The share of Platform/Tooling more than doubled from ~5.2% to ~12.1% of PRs per person and Frontend/UI emerged as an entirely new task. We are now seeing economists building full-stack web applications, something that would previously have required collaboration with separate engineering teams. Put differently, the data shows a redistribution across task types that is driven by differential shifts in productivity: the categories where AI compresses costs most lose share, while newly feasible categories absorb the freed capacity.

This raising of minimum viable competence in several new classes of tasks is particularly interesting as it allows teams like ours to avoid certain transaction and coordination costs. In projects that require a highly diverse set of skills (e.g., building a UI for a budget allocation system), a lot of time is spent on describing what’s needed to different functions, generating alignment, getting on a roadmap. AI massively lowers the self-production costs in such scenarios. When the cost of doing it yourself with AI falls below the cost of delegating to a specialist plus the coordination tax, the task migrates. This applies both across teams (scientists absorbing frontend work) and within teams (individuals becoming more self-sufficient). The direction of task migration depends on where AI’s skill shift bites hardest: codifiable tasks are most susceptible to absorption. Frontend/UI is the clearest case of such task absorption for our team. In 2023, virtually no economist wrote frontend code. By 2025, economists were building full-stack web applications. Previously, such work would have required delegating to separate engineering teams with all the attendant coordination costs: getting on their roadmap, describing requirements, iterating on designs.

Importantly, it is not the case that economists are encroaching upon or replacing frontend or platform engineering. The applications being built are internal tools and tooling that would likely never have made it onto those teams’ roadmaps in the first place. However, AI has effectively unlocked a class of previously unfunded work, allowing us to self-fund and self-execute projects that previously fell below the line for any team to own due to their required diversity of skills.

A concrete example is Apex, an experimentation dashboarding and NPV-grounded decisioning tool built end-to-end by the Economics team. Apex was scoped to fill a real gap in our stack to enable easier cross-experiment monitoring of key metrics to drive faster and better decision-making. Spinning up a cross-team project for this in advance would have been hard to justify: the use cases were unproven and the requirements were fuzzy. AI changed this calculus. With self-production cheap enough, we shipped a working end-to-end version, put it in front of users, and let real adoption decide which pieces deserved further investment. The pieces that proved their value have since started to graduate into the central experimentation platform.

What about the large increase in Experimentation (+45.2%)? Not every category tells a clean story, and with a relatively small team spread across eight task categories, some variance is expected. The cleaner signal is in the aggregate: a broad shift away from Core ML toward Systems Engineering and newly feasible task types. Individual category movements, particularly in smaller buckets like Experimentation and Infra/DevOps, should be interpreted with that in mind.

A Pitfall: Platformization Tensions

With implementation costs dropping, it becomes particularly tempting to think about platformizing certain types of solutions. By platformization, we are referring to the abstraction of similar tasks into a codified and standardized solution. For repeated tasks, building a platform solution may be worth it when `Fixed Cost < Expected Future Uses × Manual Cost Savings per Use`. While AI lowers the Fixed Cost, it also lowers the per-instance manual cost. The net effect is ambiguous and depends on which cost falls faster.

Returning to the category composition chart, the Platform/Tooling line tells a clear story: its share of PRs roughly increased by 132%, from ~5.2% to ~12.1%. This is consistent with AI lowering the fixed cost of platform-building faster than it lowered bespoke execution costs in some areas for our team. However, this only tells part of the story and our own experience illustrates the tension between the two cost types. Early in the AI wave, we invested into a Causal Inference Platform, CIP. CIP was a UI-based tool that provided company-wide access to standard causal inference methods, such as synthetic controls. This seemed like the right platformization bet: automate repeated analytical workflows behind a clean interface.

But AI may have undermined the premise. As AI coding assistants made bespoke causal inference dramatically faster to execute, the per-instance savings from a standardized platform shrank. Rather than clicking through the UI and being constrained by our design choices, users can simply type “Use a synthetic control to estimate the impact of our investments in region A on GTV.” The agentic interface can leverage our internal MCPs to explore annotated data tables, write Python estimation and visualization code, and run the analysis with real data in one shot — with the user able to adjust and guide in ways that are not possible in the UI tooling. The complexity of real causal inference fit poorly into a constrained UI, and contrasts with Apex where context, analysis, and decision frameworks are more tightly controlled.

For some use cases like causal inference, the rise of AI-powered skills and agents suggests a different model entirely: machine-interactable tools, such as MCPs and Agentic Skills, that an AI can invoke flexibly, rather than human-interactable dashboards that lock in a particular workflow. Machine-interactable tools can still provide vetted standardization of causal inference techniques (e.g., leveraging synthetic difference-in-differences as the default rather than vanilla synthetic controls or conducting specific falsification tests to verify key methodological assumptions), but also enable the end-users to use the tools in the agentic environments they are already doing data analysis within and with greater flexibility. We may have gotten the form of the platform wrong even as we correctly identified the impulse to platformize.

Organizations like ours will have to think thoroughly about which tasks belong in which roles and which workflows deserve platforms, and how those platforms should be constructed in an agentic-first work environment.

Conclusion

The analysis of our team’s GitHub data reveals patterns that are broadly consistent with theoretical arguments from the literature. The emergence of AI seems to have caused heterogeneous shifts in the distribution of skills, which in turn have meaningfully changed the types of tasks we work on. Where the productivity impact has been largest, cost-per-output has dropped and capacity was freed; where the capability floor rose, entirely new categories become feasible. For the team, this meant that a lot of exciting new areas opened up for exploration, adding entirely new facets to what we have historically worked on.

Looking ahead, we expect more changes to come our way.

AI-driven shifts in the industry may also alter what is demanded of applied scientist teams, such as increased prioritization of cutting edge methodology that improves experimentation velocity to keep up with the pace of new feature development.

And, paradoxically, the tasks that entered our portfolio most recently due to a lowering of the capabilities floor may also be the ones that exit again for the same reason. As AI’s cost on these tasks continues to fall, theory would predict a second reallocation wave towards judgment-intensive work where the productivity gap is largest — problem framing, modelling, causal identification, deciding which results to trust. We are beginning to see exciting new directions opening up in this domain as we try to establish effective collaboration models with agentic partners.

Thank you for reading! This post is part of a series covering the Economics Team at Instacart and the areas we work on. If you would like to learn more about our work, be sure to check out our other posts on optimization via regression discontinuity designs or on using surrogate indices for measuring long-run treatment effects. You can also follow tech-at-instacart to be notified as we release new posts.

How AI Changes the Role of Applied Scientists was originally published in tech-at-instacart on Medium, where people are continuing the conversation by highlighting and responding to this story.

Background: Marketing Across Marketplace and Storefront

Instacart operates across two distinct commerce experiences:

• Instacart Marketplace, our first-party consumer marketplace
• Storefront Pro, our white-label e-commerce platform for retailers

For years, our marketing automation infrastructure was built primarily to support Marketplace use cases. That model worked well in a first-party environment, where the product experience, customer relationship, and brand were all centrally managed by Instacart.

Storefront Pro introduced a very different set of requirements. As the platform scaled to more than 350 retailers, we needed to support hundreds of independent brands, each with its own brand identity, customer base, and marketing strategy. Retailers wanted the same level of personalization and lifecycle marketing sophistication that is available on the Instacart Marketplace, but in a way that preserved their own brand and operational independence.

That raised a core architectural challenge. How do we deliver Marketplace-grade personalization and lifecycle marketing capabilities to hundreds of retailers without sacrificing tenant isolation, performance, or ease of use?

To succeed, the platform needed to let retail marketers:

• Launch onboarding, winback, and promotional campaigns
• Customize branding and messaging
• Target specific customer segments
• Measure performance and iterate quickly

At the same time, we could not simply extend a single-tenant marketing system to a multi-tenant environment without introducing serious risks, including:

• Cross-retailer data leakage
• API rate-limit bottlenecks
• Manual operational overhead
• Inconsistent brand experiences
• Tight vendor coupling

Solving those constraints required rethinking the architecture so that it could support self-service marketing at scale while preserving the isolation and reliability each retailer expects.

Architecture Overview

Our solution builds on a third-party marketing automation platform, while adding critical multi-tenancy and scaling capabilities that weren’t available out-of-the-box.

Third Party API

At Instacart, we leverage a third party vendor to facilitate marketing campaigns across a variety of channels such as Email and Push Notifications. Historically, this has been limited to marketing for the Instacart Marketplace. To extend this functionality for our white-label retailers, we expanded our relationship with the vendor, building on an existing foundation and familiarity with the platform.

We provision a dedicated workspace for each retailer within the third-party provider. A workspace is an isolated account with its own customer data, templates, and configuration. This per-retailer workspace model guarantees data isolation and brand integrity across the platform.

Managing Workspaces at Scale

Traditionally, Instacart marketers leveraged the third-party UI to manually manage campaigns & workspaces. However, this does not scale to the hundreds of Storefront Pro Retailers on the Instacart Platform.

Rather than exposing retailers to a separate system, we built tooling within Instacart’s existing retailer platform. This keeps marketing configuration in the same place retailers already manage their business and ensures a consistent, streamlined experience.

Key Components

To make the architecture more concrete, it helps to walk through the system from the retailer marketer’s point of view all the way to message delivery.

1. Instacart Tooling for Retailers

Retail marketers start in an Instacart-built React interface. From there, they can configure supported campaign types, customize template variables such as subject lines, images, offers, and disclaimers, choose their target audience, preview email and push content in real time, and launch campaigns.

This layer gives retailers meaningful flexibility while still enforcing the guardrails needed to keep campaigns consistent, supported, and safe to operate at scale.

2. Campaigns Engine

Once a campaign is launched, the Campaigns Engine handles audience evaluation and personalization setup. It determines which customers match the target audience, assigns each customer to an experiment variant, generates any associated offers, and emits one event per customer to our streaming platform.

Because this engine is already used across both Instacart Marketplace and Storefront, we can reuse proven targeting and experimentation logic instead of creating separate systems for each business.

3. Stream Consumer & Batching Layer

The Campaigns Engine produces events at the individual customer level, but processing each one independently would create unnecessary overhead downstream. To solve that, we added a stream processing layer that consumes those events, batches them into groups of up to 50, and forwards the batched requests to the CRM Service.

This batching layer significantly improves throughput and helps reduce pressure on downstream APIs.

4. Instacart CRM Service

The CRM Service is responsible for preparing and sending the messages. It validates idempotency, routes each request to the correct retailer workspace, assembles personalized content, and integrates with the third-party provider.

Just as importantly, this service isolates provider-specific behavior behind a clean abstraction layer. That gives us flexibility to change providers in the future, support multiple providers at once, and avoid tightly coupling our core platform to any one vendor.

5. Third-Party Provider Workspaces

At the final stage, messages are sent through isolated third-party workspaces for each retailer. This model ensures customer data stays separated, templates remain brand-specific, rate limits are managed independently, and each retailer’s brand identity is preserved throughout the delivery flow.

From there, messages are delivered to end customers through channels such as email and push notification.

Solving Scale Challenges

Supporting multi-tenant marketing at scale meant solving for both volume and efficiency. Large retail campaigns can involve hundreds of thousands, and in some cases millions, of personalized messages. Each message may include retailer-specific branding, customer-level experimentation, and dynamically generated offers.

That scale introduced an important systems challenge: how do we preserve deep personalization without creating bottlenecks in the delivery pipeline?

Batching for Throughput and API Efficiency

One of the core constraints came from the shape of our internal campaign pipeline. Instacart’s Campaigns Engine emits one event per user after audience evaluation and personalization setup. Left as-is, that would require the CRM Service to process users one at a time, creating unnecessary network overhead and placing avoidable pressure on downstream systems.

Those costs mattered because our third-party provider imposes strict API constraints. For example, requests are rate-limited per retailer, and individual send APIs support batches of up to 50 users per call. Processing users individually would have made large-scale campaign delivery both slower and more expensive.

To address that, we introduced a stream consumer that rebatches customer-level campaign events before handing them off to the CRM Service. Instead of processing one user per request, the system groups users into batches of up to 50 and sends them downstream together.

This change significantly improved throughput, reduced API pressure, and aligned our internal processing model with the capabilities of the provider.

Horizontal Scaling and Reliable Delivery

Batching improved efficiency, but we also needed the system to remain reliable under sustained load.

The CRM Service is implemented as a Rails engine backed by asynchronous Sidekiq workers distributed across multiple nodes. When the stream consumer forwards a batch, the CRM Service first validates idempotency to account for at-least-once delivery semantics in the streaming layer. It then enqueues asynchronous jobs to prepare personalized messages and send them through the appropriate retailer workspace.

This model gives us two important advantages. First, it keeps the ingestion path lightweight by offloading the heavier personalization and delivery work to background jobs. Second, it allows us to scale horizontally by adding worker capacity as campaign volume increases.

We also persist and send results in a dedicated datastore, which gives us a durable record of message delivery and helps support observability, retry handling, and operational debugging.

Operational Considerations

Domain Configuration

As part of sending emails on behalf of our whitelabel retailers, we needed to consider the “from address” that users would see in their inbox. How we modelled this became important, as it also has an impact on the IP warming (see below).

In order to streamline this process, we acquired an instacart-agnostic domain (eg. example.com). For each retailer onboarding to the offering, we configure an email address of retailerName@example.com which we can manage and automate internally, rather than relying on the retailer to make any DNS changes.

IP Warming

Common to the industry, launching email campaigns for new retailers requires gradually warming IP addresses to maintain sender reputation.

We automated this process to the following:

1. Start with small daily send volumes (50–1,000 emails)
2. Gradually increase over 4–6 weeks
3. Monitor bounce rates, spam complaints, and deliverability metrics
4. Automatically adjust send volume based on metrics

To use infrastructure efficiently, we share IPs across retailers where appropriate. The system monitors deliverability signals — bounce rates, spam complaints, and engagement metrics — and dynamically adjusts send volume or triggers capacity expansion when thresholds are reached. This keeps us from over-provisioning while ensuring we can scale smoothly as campaign volume grows.

Observability

Because the platform spans campaign configuration, event streaming, asynchronous processing, and third-party delivery, observability is critical to operating it reliably.

We instrument the system at multiple levels. At the campaign level, we track send rate, delivery success, opens, and clicks. At the system level, we monitor API latency, error rates, queue depth, and worker utilization. At the business level, we capture metrics such as revenue attribution, cost per send, and campaign ROI.

These signals feed into both Datadog and Snowflake. Datadog supports operational dashboards and alerting, while Snowflake powers downstream reporting and helps surface performance insights back to retailers.

Resilience and Recovery

We also built the platform to handle failures gracefully. In a distributed, event-driven system, retries and downstream instability are inevitable, so reliability mechanisms need to be built into the core flow rather than added later.

To support that, the platform uses idempotent message identifiers to prevent duplicate sends during retries, throttling controls to protect shared resources under load, and campaign pause mechanisms that allow problematic sends to be halted quickly. We also isolate failures at the workspace level so issues affecting one retailer do not cascade across the broader system.

Together, these operational safeguards help ensure the platform remains reliable even as campaign volume, retailer count, and delivery complexity continue to grow.

Template Tooling for Retailers

Content Creation at Scale

Managing email and push notification templates across 100+ retail brands presents a unique scaling challenge. Each retailer needs brand-specific content while maintaining consistency in campaign structure and quality. Our solution leverages automation and standardization to make template management efficient and scalable.

We maintain a standardized library of email and push notification templates written in Liquid, a templating language, within an internal retailer email management repository. These templates are organized by lifecycle stage and support multiple campaign types out of the box.

Template deployment is fully automated through our CI/CD pipeline. When marketers merge template changes or onboard a new retailer, our deployment scripts automatically:

1. Parse a template metadata file to determine which retailers need updates
2. Interface with Instacart’s CRM Service to generate retailer-specific versions with the appropriate branding variables
3. Upload or update templates across all relevant retailer accounts via API

This automation eliminates the manual overhead of managing templates across dozens of retailer accounts. What previously required hours of manual work — logging into each workspace, uploading templates, verifying configurations — now happens automatically in minutes.

The pipeline also supports retailer-specific customizations. If a retailer only wants a subset of our template offerings or requires custom modifications, we simply update their configuration in the metadata file. The deployment system handles the rest, ensuring each retailer receives exactly the templates they need.

Self-Service Template Customization

While automation handles template distribution, retailers still need the ability to customize content for their brand voice and marketing strategy. We built a self-service template editor that balances flexibility with guardrails.

Each standardized template is designed with configurable variables — subject lines, content copy, images, promotional disclaimers, etc. — and provides sensible default values for each. Retailers can customize these variables to match their brand voice and marketing strategy, with their values injected at send time by our campaigns engine.

Instacart’s tooling for retailers provides an intuitive React-based interface where retail marketers can:

• Customize each base template’s content and styling by overriding template variables
• Configure campaign-specific offers and incentives
• Customize branding elements pulled from the retailer’s internal asset library
• Preview changes in real-time across both email and push notification formats

The live preview capability is particularly valuable — retailers can see exactly how their customizations will appear to end users before launching campaigns. This reduces iteration cycles and builds confidence in the final output.

Behind the scenes, when a campaign launches, our campaigns engine merges these retailer-specific configurations with the base template, effectively overriding default values while maintaining the proven template structure. This approach gives retailers creative control while ensuring technical consistency and deliverability across all campaigns.

Results and Impact

The platform successfully launched and is now powering marketing campaigns for multiple retail brands:

• Hundreds of thousands of personalized messages sent per campaign
• 99.9% delivery success rate maintained across all retailers
• Sub-minute template updates from editor to production
• Reduction in operational costs through optimization
• Zero cross-retailer data leakage incidents due to robust isolation

Perhaps most importantly, retail partners can now self-service create and launch campaigns without engineering involvement, dramatically reducing time-to-market for new marketing initiatives.

AI-Driven Optimization and What’s Next

One of the biggest advantages of this architecture is that it gives us a strong foundation for future optimization. Because campaign configuration, audience evaluation, message generation, and delivery are already separated into clear layers, we can improve individual parts of the system without having to redesign the entire platform.

Adaptive Campaign Optimization

Today, retailers can launch highly personalized campaigns at scale. Over time, we see an opportunity to make those campaigns more adaptive as well.

Rather than relying only on pre-launch configuration, we are exploring ways to use early campaign performance signals to improve outcomes while a campaign is still running. For example, the system could monitor initial engagement, compare variant performance, and adjust subject lines, offers, or creative treatments based on what is working best.

This would introduce a tighter feedback loop into lifecycle marketing and help retailers optimize for outcomes such as engagement, conversion, or revenue.

AI-Assisted Content Generation

We also see an opportunity to reduce the manual effort required to create effective marketing content.

We are prototyping tools that can help marketers generate subject lines aligned to a retailer’s brand voice, suggest stronger copy based on historical campaign performance, and surface likely improvements before a campaign is launched. Over time, these tools could also help predict open rates, click-through rates, or identify segments that may need a different message strategy.

The goal is not to replace marketer judgment, but to help teams move faster and make better-informed decisions.

Multi-Channel Intelligence

Longer term, we believe the same platform can support more coordinated decision-making across channels.

Because the architecture already abstracts provider integrations and relies on an event-driven pipeline, it positions us well to expand beyond single-channel execution. That could include cross-channel orchestration across email, push, and SMS, send-time optimization at the customer level, and smarter allocation of messaging volume across campaign types and objectives.

As we continue investing in the platform, these capabilities can be layered onto the existing foundation without requiring a full re-architecture.

Conclusion

Building a scalable, multi-tenant marketing platform required solving complex challenges around data isolation, performance, usability, and operational efficiency. By thoughtfully architecting around vendor limitations and prioritizing self-service capabilities, we transformed marketing from a manual, engineering-heavy process into a robust, automated platform.

The result is a system that enables personalized, brand-specific marketing at scale — driving engagement and conversions for multiple retail brands while maintaining complete data isolation and operational excellence.

Acknowledgments

This project involved contributions from teams across Growth Engineering, Infrastructure, Marketing and Product. Special thanks to the team members who drove technical design, implementation, and launch: Ryan Martin, Brent Scheibelhut, Shradha Menon, and many others who contributed along the way.

Scaling Personalized Marketing for Multi-Tenant Commerce Platforms was originally published in tech-at-instacart on Medium, where people are continuing the conversation by highlighting and responding to this story.

Authors: Trey Zhong, Xiyu Wang

Contributors: Joseph Haraldson, Sharad Gupta, Sarah Lamacchia

Introduction

Carrot Ads is Instacart’s omnichannel retail media solution that allows retailer partners to build and scale their own advertising businesses on either their owned-and-operated (O&O) websites and apps or their whitelabel Storefront hosted by Instacart. Carrot Ads empowers retailers and CPG brands to accelerate revenue, while improving the customer experience, engagement and Ads return on investment. It features enterprise-grade infrastructure, AI-powered optimization, years of proprietary first-party data and flexibility to choose from retailer-sourced Ads demand, Instacart-sourced demand from 7,500+ CPG brands, or both.

However, onboarding a new partner onto Carrot Ads introduces a key challenge: the ‘cold start’ problem, where limited historical interactions make it difficult to predict user behavior accurately.

To serve performant ads, our systems rely on predicting a user’s Click-Through Rate (CTR) to generate a ranking score. On the Instacart Marketplace, we have billions of historical signals to train a model to do so. But when a partner launches a new ads experience on their O&O e-commerce site, there is often little to no interaction history for that property, so training an accurate model becomes challenging. User behavior can vary dramatically between websites — for example, browsing patterns on a grocery site differ from those on a pet supply or electronics site.

Training a model from scratch for a new domain is data hungry. Conversely, directly deploying Instacart’s existing Marketplace model often fails to capture the nuances of the partner’s specific inventory and user base.

To address this, we developed a Domain Adaptive Learning approach that transfers knowledge from Instacart’s data-rich environment to new partner environments. By treating the Instacart Marketplace as a source domain and the partner’s website as a target domain, we can transfer knowledge to bootstrap performance with a relatively smaller amount of data. We also found that even when there is enough data to train a model directly on the target domain, the domain adaptive model still performs better because of the benefits from Instacart’s first party data.

Domain Adaptive Learning

What is Domain Adaptive Learning?

At a high level, Domain Adaptive Learning is a subset of transfer learning. It focuses on transferring knowledge gained from solving a problem in a data-rich environment (source domain) to improve performance in a related, often data-scarce environment (target domain).

Instead of initializing a new model with random weights for every partner, we reuse representations and relationship signals learned from Instacart marketplace data to “warm start” the model. This saves labeled data and computational power, but more importantly, it allows us to deploy performant models in scenarios where the target domain lacks sufficient history to converge on its own.

Benefits & Challenges

The benefits of domain adaptive learning are significant.

• Performance:
  - In Low-Data Scenarios: Allows models to perform well, even with limited labeled data in the target domain.
  - In High-Data Scenarios: Improves performance beyond what models trained solely on the target domain can achieve.
• Efficiency: Drastically reduces training time and development costs for new domains by reusing pre-trained components.
• Generalization: Enables robust models that can generalize well in different but related domains, even when there are distribution shifts.

Model Architecture

The Domain Adaptive Learning method is based on a wide and deep Predicted Click-Through-Rate (pCTR) model architecture commonly used in large-scale recommendation systems. This model predicts CTR by first transforming raw inputs, like user IDs and product text, into dense feature embeddings. These features are concatenated and processed through two parallel paths: an interaction layer for learning explicit feature interactions and a deep Multi-layer Perceptron (MLP) tower for learning complex, hidden patterns. The outputs are then merged and passed through a final MLP to synthesize the findings. Finally, a Sigmoid activation squashes the result into a probability score (pCTR) between 0 and 1. This architecture combines a linear “wide” model (for memorization of specific feature interactions) with a “deep” neural network (for generalization). More details about this architecture can be found at this other blog post.

Our strategy for domain adaptation occurs at two distinct layers: the Neural Network level and the Training Data level.

Domain Adaptation At Neural Network Level

In Instacart Ads’ pCTR model, transfer learning at the neural network level involves reusing and fine-tuning components from a pre-trained model that originated from a related domain or task. Specifically:

1. Shared Embedding Layers: The model utilizes embedding representation layers pre-trained on shopping contexts. These embeddings capture fundamental signals that are transferable.
2. Feature Transfer Domain Adaption: The model structure allows seamless integration of pre-trained embeddings with domain-specific input features. For instance, “Wide” components might focus on explicit features (e.g. historical CTR for a product category) sampled from the new domain, while “Deep” components adapt pre-trained dense representations.
3. Fine-Tuning Specific Layers: While shared layers are reused without major alterations, subsequent layers are fine-tuned using limited partner-specific training data to capture domain-specific behavior.
4. Generalization: Transfer learning ensures the model can generalize knowledge learned from user interactions in Instacart Marketplace to predict user responses in the partner’s domain. This prevents the need to train the deep ranker entirely from scratch.

Domain Adaptation At Training Data Level

Transfer learning at the data level involves aligning the input signals of the source and target domains so the model “speaks the same language.”

We rely on aggregated historical performance signals to normalize features across domains, but there are a variety of details that contribute to the quality of our training data.

• Source Data: Large-scale data from Instacart Marketplace’s user behavior is leveraged as the source domain. This data is used to pre-train embeddings and build a foundational model.
• Matching Features Between Domains: Common contextual and catalog-level features between the Instacart Marketplace’s catalog data and the Carrot Ads Partner’s catalog are aligned (e.g. ensuring product category uses the same taxonomy) to ensure the source domain knowledge is transferable.
• Feature Trimming for Latency Optimization: To meet real-time auction latency requirements and be flexible to various feature availability for the partners, we apply feature trimming technique to balance performance and speed. We analyze feature importance in the target domain and prune inputs that do not contribute to prediction accuracy for that specific partner, ensuring the model remains lightweight.

Learnings & Conclusions

Our evaluation of Domain Adaptive Learning demonstrates that it is possible to achieve satisfactory pCTR prediction accuracy with limited data from our partners. By leveraging the “source” knowledge of the Instacart Marketplace, we achieved higher CTR, total clicks per user and ads revenue across search ads and product category ads. This approach enables us to launch high-performing ad networks for partners immediately, eliminate the traditional data ramp-up period and converge to a better stable state.

However, this process is not yet fully autonomous. The complexity of mapping data schemas and verifying model alignment currently requires human-in-the-loop verification to prevent negative transfer.

Looking ahead, we are building an automated Domain Adaptation Platform that can detect domain shifts and fundamentally streamline the workflow. This allows us to onboard new retail partners faster and in a more scalable way, while continuing to deliver performant ad systems from day one.

References

• One model to serve them all

Empowering Carrot Ads with Domain Adaptive Learning was originally published in tech-at-instacart on Medium, where people are continuing the conversation by highlighting and responding to this story.

Key Contributors: Moein Hasani, Hamidreza Shahidi, Trace Levinson, Guanghua Shu

Introduction

At Instacart, we are laser-focused on improving the user experience by making shopping feel easy, engaging, and personalized. Our discovery surfaces play a central role in bringing this to life. Alongside explicit Search intents, discovery is our opportunity to meet customers’ implicit needs, presenting them with the most relevant and inspiring content we have to offer. The main discovery surface within the Instacart app, referred to here as the “Shopping Hub”, is one of the most critical in this regard. This is the surface a customer lands on within the Instacart app after selecting their desired retailer, guiding them along their entire journey. What users see here shapes not just what they buy, but how intuitive and enjoyable their experience feels.

Given its importance, our team runs dozens of Shopping Hub experiments per year, constantly evaluating new ways to enrich the discovery experience. Historically, these experiments have been constrained by static content libraries feeding our recommendation systems.

With the rapid advancement of generative AI, a critical opportunity began to emerge: rather than incrementally improving a swath of legacy systems, could we leverage LLMs to rethink how content shows up for a user from the ground up? Which new primitives could we build to uplevel quality, personalization, and cohesion across the page?

This blog post walks through our early journey to answer these questions. By investing in a new AI-native platform for content generation, evaluation, and retrieval, we have found generative models to show real promise in improving recommendations at scale. Below, we highlight the approach we took in developing this platform, a few key learnings so far, and where we’re most bullish moving forward.

Limitations of Traditional Recommendation Engines

Our Shopping Hub page is constructed from multiple subcomponents called placements. Each placement contains a number of products or other entities within it. The example below can help us visualize how these various pieces ladder up to the full page.

Today, the flow to generate and serve Shopping Hub content looks something like the following:

1. Text, visual assets, and the underlying retrieval sources for a new placement are defined explicitly. Titles are often generic, such as “Dairy”, with a narrow set of retrieval sources used to fetch corresponding products. Once created, each placement enters the content library and becomes eligible for serving, universally, across all users. This generation process is human-driven and inherently assumes a one-size-fits-all content library that applies to all users.
2. Retrieval systems fetch these candidate placements and corresponding products at runtime.
3. From the retrieved set, ranking models order products and placements across the page to optimize against a static set of business metrics. Each placement is treated as an independent entity on the page within ranking models.

This setup can perform well for optimizing average engagement under the above constraints. However, the reality is that different users have different needs on the platform, and business objectives and the broader environment are constantly changing. This results in a couple of key limitations for our recommendations:

1. Difficulty in scaling personalized content: The human-driven process above is expensive and time-intensive, with teams managing both generation and content QA by hand. As a result, traditional architectures inhibit the ability to quickly deploy and personalize new content — not only per user, but also according to seasonal and other shifting dimensions and business objectives.
2. Lack of cohesion: Placements are often created by different siloed teams with divergent focus areas and goals. As a result, the series of placements can result in a chaotic surface presentation. Users are required to scroll without the ability to easily navigate the page to solve their needs.

So, where does AI come in?

Large language models offer a natural mechanism for producing cohesive, dynamic, and personalized output. We began to explore ways to tackle the above limitations by introducing generative models into our recommendations stack. To narrow down our approach, we first began thinking through objectives for the system to ensure any solution would be anchored to North Star principles.

• Delightful Personalization: The system should be capable of leveraging our rich user data to meet users where they are on the platform. One user may be health-conscious and focused on soups and salads. Another user looks to Instacart for home improvement goods and other category needs beyond only food and drinks. Given the diverse reasons our customers turn to Instacart, our primary motivator for the work was to enable rich, delightful experiences for every customer on our platform, rather than solving for averages. This may also include very different products retrieved for the same intent — a thematic “Breakfast” placement may prioritize waffles and pancakes for one user, but granola and yogurt for the other.
• Cohesion: The system should enable full cohesion across the page — every placement should be intentionally grouped, ordered, and aware of others around it. We want the discovery journey to feel seamless.
• Adaptability: The system should be responsive to rapid adjustments as our business environment shifts. This includes support for varying business objectives, such as relevance versus novelty, as well as temporal dimensions such as seasonal winter placements that can be spun up dynamically as they become relevant, then phased out.

With these guiding principles, we narrowed consideration down to two core generative paradigms:

1. Bottoms-up generation: Directly generate all possible products to serve to a user, then cluster and organize them into placements.
2. Top-down generation: Begin by generating ordered placements to structure the entire page, then generate products per placement.

To visualize this distinction, let’s take a simple example and assume we are building two placements to recommend to a user. Generative models can compose the problem in one of two ways:

In the bottoms-up approach, a model generates a raw sequence with all relevant products, then clusters those products into Breakfast Staples and Health-Conscious Snacks themes.

Under the top-down approach, Easy Pasta Night and Gourmet Salad Fixings themes are first generated and ordered to meet user relevance, cross-theme cohesion, and other business goals. Products are then generated to map to each theme.

Comparing the two methods for our use case

The bottoms-up approach contains interesting benefits, such as deep flexibility with less constrained recall. However, it also presents difficulties in real-world settings due to latency requirements and catalog turnover. Further, with a much broader modeling task, it can be difficult to ensure generated products meet a diverse set of page requirements and intents, and may require significant fine tuning efforts as needs evolve. In other words, while the first two tenets could be achieved, we felt our adaptability goal would be put at risk. To best balance personalization, cohesion, and adaptability, we landed on a top-down, cascaded approach.

Methodology

After landing on the top-down approach, our next question was how exactly to decompose the problem. In early explorations, we evaluated the possibility of an all-in-one model that would directly generate placement content from raw signals. We started with this approach for simplicity, but ultimately found great value in decomposing generation into multiple targeted tasks. This opened the door to using retrieval‑augmented generation (RAG) and other techniques that aren’t feasible in a single‑step model, enabling us to achieve higher quality while improving cost efficiency.

Overview

The system consists of a few main phases:

1. Page design & theme generation
2. Retrieval keyword generation
3. Quality and diversity filtering
4. Product and pagewise ranking

The first three phases form our generative content pipeline, while the final phase leverages existing ranking infrastructure for scalable serving.

Over the next few sections, we’ll walk through each of these components in detail and how they ladder up to the final Shopping Hub users see on the platform.

Phase 1: Page Design & Theme Generation

First, a page design agent leverages user context (purchase history, engagement signals, and other derived preferences) to produce a set of high-level themes personalized to each user. Themes are designed to represent discrete and coherent shopping intents (for example, “Flavor builders for weeknight meals” or “Functional hydration, lower sugar”). We leverage constrained decoding with a structured schema to ensure interpretability and downstream usability.

To optimize downstream token efficiency, Phase 1 outputs both placement entities as well as a set of derived signals, such as user personas and freeform product concepts that align well with user context and placement intent. This removes the need for redundant context passthrough along each stage of the pipeline.

Phase 2: Retrieval Keyword Generation

Once generated in Phase 1, each theme is mapped to one or more retrieval-compatible descriptors (ex: search query strings, categories from our catalog taxonomy, product attribute filters). We explored various descriptor representations and ultimately found structured, taxonomy-grounded representations to perform best in upholding relevance. For simplicity, we will refer broadly to these descriptors as keywords here.

Teacher-Student Fine Tuning

To meet latency and cost requirements, we leveraged teacher–student learning: a closed-weight LLM first generates high-quality supervised data, validated on a small sample by human annotators. An LLM judge is then used to prune poor-quality data from our fine-tuning dataset, and an internal model is fine-tuned to imitate the teacher while satisfying domain-specific constraints.

Finally, we performed a number of ablation studies to converge toward the optimal student model:

• Open-weight base model explorations across the Llama and Qwen families
• LoRA adapter addition at varying ranks
• Finetuning sample size augmentation

RAG

To further improve prompt efficiency while maintaining strong precision, we incorporated retrieval-augmented generation (RAG) into the keyword generation pipeline. First, the page design LLM in Phase 1 generates freeform product concepts, such as “eggs”, from its universal knowledge base that align well with user context and placement intent. Embeddings are generated for these concepts. In the keyword generation model, we then restrict eligible candidate keywords per theme using embedding‑based similarity. Roughly 100 nearest neighbors are retrieved from a 300,000‑term keyword corpus, and only this refined subset is passed down for the second LLM to select from as final recommendations. This first-pass candidate pruning reduces input context significantly in the second LLM, reducing all-in generation costs by 15–20% in each generation. This became a core motivator for adopting a cascaded generation architecture. A single‑LLM setup would instead require the full keyword corpus to be passed directly into the prompt to maintain the same level of precision.

Phase 3: Quality and Diversity Filtering

Given the dynamic nature of this system, guardrails help to prevent cross-placement redundancies and ensure high-quality content. The system handles this in a few stages:

1. To ensure sufficient diversity, embeddings are generated for each placement’s content, and similarity-thresholded deduplication is applied to remove redundant placements.
2. For broad quality validation, LLM-as-a-judge workflows are deployed against a small proportion of users to ensure overall theme quality and brand compliance. Theme-product relevance is then enforced through a fine-tuned cross encoder, which explicitly classifies the relevance of each placement’s products to their overarching theme. Low-scoring entries are flagged for offline filtering or repair before serving deployment. Our full suite of evaluators is described in more detail in the Evals section below.
3. Finally, additional guardrails kick in to enforce business and policy constraints. For example, we should ensure all original business objectives from agent instructions are addressed. Furthermore, it is critical to ensure themes do not misalign with Instacart’s brand, or even hallucinate with harmful or inappropriate pairings like alcoholic products for a child’s birthday party.

Phase 4: Product & Pagewise Ranking

Finalized placements and keywords are cached for runtime retrieval. Existing product and placement ranking services retrieve all generated entities, perform additional ranking and post-processing, and return finalized ordered entities on the page. This design modularizes the system, decoupling generative retrieval from mature ranking systems and providing a path to deeper pagewise control as the generative component matures.

Designing for Rapid Iteration: Treating Evals as a First-Class Citizen

When developing any AI-native system, particularly one that generates dynamic content served to millions of users, quality enforcement is essential. Potential for off-brand or other low-quality content can quickly degrade trust with our customers, so our team invested deeply in designing a robust suite of LLM-based and other evaluators, enabling us to iterate with confidence. This not only helped us derisk adverse behavior; it also became a massive accelerant. Given the vast exploration space for generative recommendations, online iteration would be slow, variance-prone, and cost-prohibitive. After a temporary slowdown upfront, the benefits of our QA investments have begun to compound across both velocity and output quality.

Below, we’ll walk through our three-pronged Eval framework:

1. LLM-as-a-judge evaluators
2. Fine-tuned QA at scale
3. Traditional ML and metric-based evaluators

LLM-as-a-Judge

First, a rich suite of LLM-as-a-judge evaluators audits output along each level of the content hierarchy. Quality is graded along dimensions such as the following:

At the page level:

• Does the page feel cohesive enough? Diverse enough?
• Does the full set of generated placements cover all of our business needs?

At the placement level:

• Are the titles of high quality and aligned with our brand?
• Do placement themes align with user preferences and order behavior?

At the product level:

• Have we maintained sufficient product recall in the final output?
• Are the underlying retrieval keywords and products still aligned with the title’s thematic intent?

To build trust in this framework, we developed a series of human-in-the-loop (HITL) workflows to build ground truth data, tuning the LLMs until passing high human-alignment thresholds.

Unlocking Evaluation at e-Commerce Scale

LLM-as-a-judge evaluators are a powerful tool. However, we found that while this framework guided us well at the averages, it failed at the edges. Since evaluating millions of candidates is cost-prohibitive, LLMs are unable to take action and improve quality at scale. Certain quality dimensions hit diminishing returns, such as preserving end-to-end model context: final products retrieved did not always align well with the placement’s upstream thematic intent.

Given this insight, we made the decision to supplement Evals with a fine-tuned DeBERTa model, classifying product-title relevance for every generated placement. This model is trained on the same HITL ground truth data generated for LLM-as-a-judge evaluators, synthetically augmented for broader teacher-student model learning.

This model unlocked over a 99% cost reduction relative to closed-weight LLM inference. This enabled us to leverage it not only for evaluation, but also for full-scale quality filtering, where any placements classified as a severe violation are pruned before deploying to production.

Classical ML and Metric-Based Evaluations

Lastly, we rounded out the suite with a number of classical ML and metric-based evaluators. These span both explicit and derived signals already built for other use cases. We have found these to be useful proxies for broad relevance and quality:

• Average proportion of products represented in the user’s purchase history
• Predicted user-product engagement scores from our existing ranking models
• Average products per placement (density)

Bringing the Pieces Together

Let’s see how the above pieces come together with a more detailed view of our content generation & evaluation architecture:

Results

This generative merchandising framework leads to a meaningful shift in placement composition. A sample of static vs. AI-generated placements for the same user can be seen in Figures 2a and 2b below. Compared to rigid single-category placements, generative placements tie closely to the user’s shopping history and build engaging themes composed of several underlying categories (e.g. meats, cheeses, and breads).

Amongst dozens of iterations, we began to observe generative policies outperform our baseline in offline evaluations. This validation finally built the level of confidence needed to perform large-scale A/B experiments comparing the generative page to the production baseline. While work remains to enable us to fully overhaul our prior systems, initial results have been quite promising.

Key Learnings

Over the course of this work, we have taken away broad learnings that we hope can be valuable to other AI developers.

Keep each modeling task focused: Models perform best with a well-defined task. While all-in-one workflows are tempting when working with frontier models, we have found stronger, more easily tunable performance by decomposing into multiple tasks. This became most evident in our fine-tuning efforts — smaller models required a high level of handholding in sample data and label definitions to reach strong performance.

Evals are worth the pain: It is difficult to overstate the value Evals provided in this work. Building comprehensive and human-validated LLM judges is admittedly a daunting task when kicking off a new project. But particularly for domains with highly variable output, such as personalized recommendations, a clear quality definition helps to prevent paralysis later on. We consistently steered our generation strategy to make progress against the evaluators we had set up.

Adding structure to input and output layers meaningfully improves outcomes: A number of optimizations to our inference data flows were impactful in bringing this system to production. Efficient context handling — especially through RAG and aggressive token compression — unlocked richer input signals without ballooning cost. In the output layer, constrained generation ensures the model always produces reliable, production‑safe outputs. Together, these examples speak to a broader principle: well-structured flows lead to more dependable and scalable agentic systems.

Looking Forward

We are just getting started on this platform. Moving forward, we will expand the system to balance multiple objectives, such as relevance and novel inspiration, and introduce deeper personalization through real-time and sparse signals. We are also exploring reward modeling and reinforcement fine tuning (RFT) to enable self-improvement with tight feedback. An exciting direction here will be learning how our stack of traditional ranking models can be fused as reward models within post-training, bringing recommendations fully into the generative paradigm.

In parallel, our teams are also exploring how to scale generative recommendations to surfaces beyond the Shopping Hub, such as landing pages and Search results. Early experiments are showing promising signs — stay tuned!

Acknowledgements and Final Notes

We would like to extend deep gratitude to our cross-functional partners Dhruv Khanna, Logan Murdock, Roy Li, Aref Kashani, Shayaan Nadeem, Amish Popli, Lauren Downey, Shrikar Archak, Brett Brownell, and Brandon Silberstein, who have provided critical ongoing design feedback and driven system integrations to bring this research to production. Vinesh Gudla, Hechao Sun, Jingying Zhou, and Tejaswi Tenneti also made meaningful contributions in our early stages of development. Additional thanks to Pramod Adiddam and Venkatesh Shankar for steady leadership and support, enabling the team to push forward.

Our team is investing deeply to optimize how generative AI and traditional machine learning systems intersect. Interested in helping us advance the frontier? Our machine learning teams are hiring!

Our Early Journey to Transform Instacart’s Discovery Recommendations with LLMs was originally published in tech-at-instacart on Medium, where people are continuing the conversation by highlighting and responding to this story.

Introduction

Caper is Instacart’s AI-powered smart cart that provides customers with a fast, seamless, and intuitive shopping experience. We achieve this through computer vision and multi-sensor fusion to power accurate product recognition and effortless checkout. Delivering this experience requires Caper’s AI models to understand what truly happens in stores — the movement, intention, and decisions unfolding across every grocery aisle.

Historically, our ability to learn from production environments was limited. Even though the carts were deployed in stores, we lacked a scalable way to collect real‑world data that would allow us to rapidly iterate and improve our models. This resulted in three core challenges:

• Scalable Onboard Observability: We had little visibility into what was happening on the cart, in the stores. When something went wrong, it was hard to understand or reproduce the scenario. At the same time, each cart generates gigabytes of multimodal data, from sources such as cameras, weight sensors, and localization sensors. We needed a centralized way to capture key moments so the team could clearly understand what the cart experiences, how users interact with it, and where to improve — all while maintaining a magical user experience and minimal impact on the network.
• Data Quality and Diversity: Our models were primarily trained on manually-collected data that didn’t fully reflect the complexity of real-world stores, including lighting changes, occlusions, damaged packaging, motion blur, unusual angles, and store-specific products. This gap can hurt the user experience. We need a reliable way to systematically gather high-quality, diverse production data that could increase models’ robustness and accuracy.
• End-to-End Model Cycle Time: Turning production data into model updates required manual data cleaning, triage, labelling and training. This made the end‑to‑end model development cycle not only slow but also expensive. We want a rapid, automated way to learn from the vast diversity of real‑world data so the full end‑to‑end model iteration cycle can improve on a weekly cadence instead of every month, and so the cost of iteration would not grow linearly with deployment size.

To address these challenges, we built Capsight, an end-to-end platform that transforms our fleet of Caper carts into a distributed data collection and model improvement engine, enabling our carts to get smarter on their own over time.

The Capsight Ecosystem: Core Components

Capsight creates a continuous feedback loop connecting live edge-captured data directly to our ML training workflow. It consists of three components that work together to create our data flywheel:

Collect → Manage → Label → Train → Deploy

• Capsight Collector: An intelligent, on-device agent that captures high-value data from a variety of cart sensors (camera, weight, location, etc.).
• Capsight Depot: A centralized cloud platform for data management. It ingests, processes, and indexes all incoming data while performing data cleaning and quality processing, making the data searchable, explorable, and ready for annotation.
• Capsight Learner: A distributed training platform that consumes curated datasets from the Depot to train, evaluate, and accelerate model iteration.

Capsight Collector: Intelligent Capture on the Edge

The Capsight Collector is the foundation of the entire system: the cart’s central data-gathering agent. Its mission is to capture a holistic view of every interaction by collecting synchronized data from all of the cart’s sensors.

While the initial phase focuses on high-value computer vision data from the cameras, the Collector is designed as a multi-modal platform. We integrated other crucial data streams, such as weight data from the scale and location data within the store, to build a complete picture of each event.

To achieve this without impacting cart performance or overwhelming store networks, we solved several key engineering challenges:

• Intelligent, Trigger-Based Capture: To avoid collecting terabytes of irrelevant data, the Collector operates on a trigger-based system. It only begins capturing data when an important event is detected. The initial trigger is a combination of an activity signal (like a hand motion) and a recognized barcode, with more signals being developed. This dual signal gives us high confidence that a meaningful interaction is occurring. The sensitivity of this trigger is an important trade-off. Collecting useless data is expensive and increases noise, but missing signals decreases training input.
• Optimized On-Device Processing: We leverage dedicated hardware for video encoding, which ensures the entire collection process runs with zero performance regression on the cart’s primary AI tasks. We also built a dedicated communication protocol so that weight and location data can be collected without any performance degradation.
• Resilient Uploading Workflow: The collected data is stored locally on the cart’s storage device. The Uploader is designed to work seamlessly within the store environment, carefully managing upload timing and bandwidth to avoid any impact on retailer operations or network performance. To prevent disk space issues, it includes a storage check to pause collection if usage exceeds a configurable threshold and an auto-cleanup mechanism to remove the oldest files if the upload fails.

Capsight Depot: Ingestion, Curation, and AI-Assisted Annotation

Once the Collector uploads the raw sensor packages, the Capsight Depot begins the process of transforming these massive volumes of raw files into structured, high-quality training datasets:

• Processing: A distributed data processing system ingests the raw files, extracts metadata, and performs quality checks to ensure the data is consistent and ready for downstream use.
• Indexing, Search and Visualization: All data and metadata are securely stored, indexed, and enriched to support search and deeper semantic exploration through a user‑friendly web interface. This allows the team to quickly investigate production scenarios by filtering relevant metadata and immediately accessing corresponding videos and logs.
• AI-Accelerated Annotation and Curation: A key function of the Depot is preparing data for annotation. However, as Capsight scales to millions of images daily, manual labeling becomes a significant bottleneck in both cost and time. To break this bottleneck, we’ve integrated a powerful Vision Language Model (VLM)-based pre-labeling service directly into the Depot and Labelling Platform. Instead of sending raw images for manual annotation, the pipeline first filters out empty background images. Then, a VLM, in combination with our teacher models, automatically generates high-quality pre-labels for items and barcodes. These pre-labeled images are then sent to human annotators for rapid correction rather than slow, from-scratch creation. This AI-assisted approach is projected to reduce annotation costs by over 70% and cut down a multi-day labeling task to just a few hours. It even allows us to efficiently clean errors from our historical ground truth data.

Capsight Learner: Closing the Training Loop

With a curated, labeled dataset ready in the Depot, the Capsight Learner completes the flywheel. This distributed, Ray-based training platform automates the process of consuming these datasets to train new model versions. An automated evaluation pipeline benchmarks models against standardized test sets, ensuring only validated improvements make it to production. By providing a direct path from labeled data to a trained model, the Learner dramatically accelerates our model training cycle. As a result, we reduced the model training stage from one week to two days.

Impact: The Flywheel in Motion

By closing the data loop, Capsight has transformed our AI development process from a reactive, manual effort to a proactive, automated one. Providing a measurable and immediate impact for retailers and their customers.

We’re already seeing significant accuracy gains. Within just weeks of deployment, we collected enough diverse, real-world data to train improved models that showed more than 5% improvement in accuracy, with continued gains as the deployment scales. Our dataset is now richer and systematically captures edge cases, lighting variations, and store-specific products that make our AI models more robust.

The iteration cycle is also dramatically faster. The full loop of collecting data, labeling, training, and releasing a model used to take about a month, and now completes in a week. Customers can benefit from these improvements much sooner.

Future Work: What Capsight Unlocks

Our journey with Capsight is just beginning. Next, we’re expanding beyond vision to full sensor fusion, combining camera, weight, motion, and location data. This richer, multi-modal dataset will fuel a foundation model capable of understanding real-world store environments across vision, motion, weight, and behavior.

This foundation model enables powerful capabilities:

• Detecting complex multi-item interactions and intent
• Improving location-based experiences
• Automatically surfacing the most valuable data for model improvements

As the system scales, we’re also optimizing costs: from multi-attribute extraction in a single pass to efficient VLM inference, ensuring that Capsight grows smarter and more scalable with every iteration.

Conclusion

Capsight represents a step change in how we build and ship AI with Caper at Instacart. By wiring observability and data feedback directly into our ML loop, we’ve created a closed-loop system that accelerates learning, boosts model accuracy, and strengthens reliability across our in-store retailer technology.

With higher accuracy, faster debugging, and dramatically shorter iteration cycles, Capsight allows retailers and their customers to feel improvements as soon as they’re made. It transforms real‑world data into a continuous innovation engine — one that strengthens Caper today and lays the foundation for the next generation of in‑store AI.

Turning Data into Velocity: Caper’s Edge and Cloud Data Flywheel with Capsight was originally published in tech-at-instacart on Medium, where people are continuing the conversation by highlighting and responding to this story.