SATU: Technical Devpost

TechJam 2026, Problem Statement #4: Shopping Copilot

Repository: https://github.com/justin-theodorus/techjam

Technical Summary

SATU is a deterministic, stateful shopping agent for a 50,000-product catalog. It treats recommendation as a sequential decision problem: every exposed product consumes a ranking position, and every clarification question consumes a turn.

The system dynamically controlsDe both:

slate width: how many products to expose now
question budget: whether another clarification turn is useful

SATU runs entirely in memory using the Python standard library. It makes no external calls on the scored path.

Metric	SATU	BM25 baseline
TechnicalScore	0.9672	0.1067
HitRate@10	1.000	0.125
MRR	0.975	0.068
MTTC	2.27	9.81
Latency per turn	~1 ms (p50 0.96 ms, p95 3.2 ms)	8.5 ms
Runtime cost	$0.00	$0.00

How to read this document

The document follows the request as it moves through the system. Each section names the pipeline stage it describes, and every formula appears in the section for the stage that evaluates it:

Objective  →  Architecture  →  Turn pipeline
                                   │
                                   ├─ Understanding + state   → "Conversation State"
                                   ├─ Behavior selection      → "Dynamic Behavior Selection"
                                   ├─ Retrieval + ranking     → "Retrieval and Ranking"
                                   ├─ Slate shaping           → "Dynamic Product Exposure"
                                   └─ Question selection      → "Dynamic Question Control"

System Objective

The evaluator terminates a session as soon as the hidden product appears in the returned slate. This creates an asymmetric cost:

showing the correct product converts the session
showing an incorrect product proves it is not the target
hiding a plausible product preserves it for later reranking
asking a question is useful only if its answer can change a future slate

SATU therefore optimizes the joint objective rather than HitRate alone:

high target coverage
+ high reciprocal rank
+ low turns to conversion

The engineering objective goes beyond the benchmark. We wanted a system that could be implemented in a real shopping product: low-latency, inexpensive, explainable, resilient to missing services, and able to justify both the products it shows and the questions it asks. That is why the scored path uses catalog-derived evidence, bounded session state, deterministic fallbacks, and no external runtime dependency.

Evaluation Objective

For N sessions, target rank r_i, and first-hit turn t_i:

HitRate@10 = (1/N) Σ 1[r_i ≤ 10]
MRR        = (1/N) Σ 1/r_i             (misses contribute 0)
MTTC       = (1/N) Σ t_i               (misses are assigned turn 11)
Efficiency = clip((11 − MTTC) / 10, 0, 1)

TechnicalScore = 0.50·HitRate@10 + 0.30·MRR + 0.20·Efficiency

The 0.30 MRR term is what makes slate width a real decision rather than a formality: showing ten products immediately can improve MTTC while reducing reciprocal rank enough to lower the combined score. "Dynamic Product Exposure" below is the section that acts on this, and it argues the case from shopper behavior first, because the scoring weights and the product argument happen to point the same way.

Architecture

Startup work is done once, at construction. Nothing below the dashed line touches disk or network.

                          STARTUP  (once, ~5.6 s, ~280 MB)
50,000 catalog products
      │
      ├── coarse category buckets ──────────► used by: hard filter          (§ Retrieval)
      ├── tokenizer + BM25 statistics ──────► used by: IDF / BM25 formula   (§ Retrieval)
      │      └── regex word tokenizer + 31 stopwords, shared by index and query
      ├── popularity prior ln(1+rating_number) ► used by: blend S(d)        (§ Retrieval)
      ├── attribute vocabulary + lead lines ─► used by: offers(d), H(a), Coverage(a)
      │                                                                     (§ Questions)
      ├── rare-phrase index (df per phrase) ─► used by: Evidence(d)         (§ Retrieval)
      └── optional dense asset ──────────────► weight 0, rejected           (§ Rejected)
                          │
                          ▼
                   IN-MEMORY CATALOG
- - - - - - - - - - - - - │ - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
                          ▼
                        TURN  (~1 ms)

 message ─► parse ─► fold state ─► select behavior ─► retrieve ─► rank
                          │              │                            │
                          │              │                            ├─► shape slate
                          │              │                            │    head(t), C(m)
                          │              │                            │    (§ Exposure)
                          │              │                            │
                          │              │                            ├─► choose or suppress
                          │              │                            │    question  Q(a)
                          │              │                            │    (§ Questions)
                          │              │                            │
                          │              │                            └─► recovery check
                          │              │                                 Spent(o)
                          │              │                                 (§ Recovery)
                          │              └─► one of six behaviors
                          │                  (§ Dynamic Behavior Selection)
                          └─► typed slots, refusals, shown set
                              (§ Conversation State)
                                                     │
                                                     ▼
                                        response + session trace

Startup takes approximately 5.6 seconds and peaks around 280 MB. After initialization, a turn takes approximately 1 ms and requires no disk or network access.

On the tokenizer: SATU does have one, and it is deliberately simple. It is a regular expression word splitter plus a 31-word stopword list (submission/src/text.py), and the identical function is used to build the index and to build the query, so an indexing decision can never disagree with a lookup decision. It is not a learned or subword tokenizer, and there is no model of any kind in the scored path.

Turn Pipeline

Stage	Input	Decision or output	Detailed in
Understanding	Shopper message	Category, constraints, pivot, refusal, or exhaustion	Conversation State
State fold	Parsed turn + previous state	Updated slots, refusals, history, and shown products	Conversation State
Behavior	Current state and candidate counts	Discovery, precision, recovery, boundary, stagnation, or coverage	Dynamic Behavior Selection
Retrieval	Active category and constraints	Candidate pool	Retrieval and Ranking
Ranking	Candidate pool and query signals	Ordered candidates and confidence structure	Retrieval and Ranking
Slate shaping	Scores, turn, route, and history	Dynamic number of visible products	Dynamic Product Exposure
Probe selection	Live candidates and conversation state	One useful question or no question	Dynamic Question Control
Response	Slate, behavior, and probe	Shopper-facing message and products	Dynamic Behavior Selection

Every decision is recomputed from the latest state. SATU does not follow a fixed dialogue tree or assign one permanent persona to a session.

Conversation State

This is the first pipeline stage. Everything later in the document reads the structure built here: ranking reads the constraints and the refusals, exposure reads the shown set, and question selection reads what has already been disclosed.

Each message is converted into typed preference slots:

category = crossbody bag
feature  = imported
material = leather
color    = black

The session state also stores:

current category and candidate buckets
active constraints and their attributes
refused or exhausted attributes
products returned on previous turns (the shown set, used by exposure)
the last question asked
consecutive questions that added no information
pivots and intent overrides
previous candidate contention and decision readiness

Targeted Override

When a shopper changes one preference, SATU removes only the slots contradicted by the new statement. For example, "Actually, make it brown" replaces the active color but preserves category, material, and feature constraints.

This avoids both common extremes:

merging mutually inconsistent preferences
discarding the entire session after a local correction

Targeted override improved the overall score by 0.016 and raised intent-override HitRate@10 from 0.900 to 1.000.

Exhaustion and Refusal

SATU distinguishes "nothing more about material" from "I have no more preferences." Attribute-specific exhaustion retires only that dimension. Full exhaustion stops further probing and opens product coverage.

This distinction prevents one unanswered question from prematurely ending all clarification. It is also load-bearing downstream: full exhaustion is one of the three conditions that releases the withheld slate, in "Dynamic Product Exposure".

Cross-Session Memory

Only low-risk signals carry across visits:

refusals
category affinity
attributes that should not be asked again immediately

Historical signals decay by 0.7 per visit. Positive product preferences do not automatically carry because a new visit may represent a different shopping mission. What we can and cannot claim about this component is stated under "Limitations".

Retrieval and Ranking

This stage turns the state above into an ordered candidate list. The next stage decides how much of that list to reveal.

hard category filter
→ BM25 relevance
→ popularity prior
→ negation penalty
→ rare-phrase promotion
→ optional reranking seam  (rejected tiers plug in here)

Category Filtering

The hard filter reduces the full catalog to a median of 182 candidates while retaining the target 99.0% of the time. This makes later ranking faster and more precise than applying all signals globally. It is also why the rejected dense retriever had nothing left to win; see "Dense Retrieval (Rejected)".

Lexical Relevance, Popularity, and Negation

BM25 responds to explicit shopper language; the popularity prior provides a stable fallback when the query is sparse or uses different wording from the catalog. The two are combined on one normalized scale.

SATU uses BM25 with k₁ = 0.6 and b = 0.3:

IDF(q) = ln(1 + (N − df(q) + 0.5) / (df(q) + 0.5))

BM25(d,Q) = Σ IDF(q) · tf(q,d)·(k₁+1)
                         ─────────────────────────────
                         tf(q,d)+k₁·(1−b+b·|d|/avgdl)

tf and |d| here are counted over the tokens produced by the shared tokenizer named in the architecture diagram, which is why an index statistic and a query term always agree.

Moving from textbook BM25 parameters (1.2, 0.75) to (0.6, 0.3) improved TechnicalScore by 0.030. The shorter catalog documents need less term-frequency saturation and less length normalization.

The normalized lexical score is blended with the normalized popularity prior:

S(d) = BM25(d,Q) / max BM25 + 0.6 · Popularity(d) / max Popularity

Popularity(d) is not a stored score. The catalog holds one raw field per product, rating_number; the prior is its log, taken once at load:

Popularity(d) = ln(1 + rating_number(d))

The log is the whole transform, and it is what keeps a 13,000-review product from outweighing a 400-review one by two orders of magnitude. Both terms are divided by the maximum over the live pool, so each lands on [0,1] and 0.6 is a true mixing weight. The same prior also orders every candidate bucket, so the stable sort leaves lexical ties resolved by popularity rather than by catalog position.

S(d) is the score the exposure rule reads. When the next section counts "products within a margin of the leader," this is the quantity being compared.

A refused attribute is then subtracted from the same normalized scale:

S(d) ← S(d) − 0.5 · Refused(d) / max Refused

Refused(d) is d's lexical score against the refused tokens, which are also removed from the positive query, so a refusal is never counted as evidence for the thing refused. A penalty rather than a filter, because refusal detection is lexical: the strings no and non occur inside 431 valid catalog attributes, so the cue list stays narrow and a false positive costs a few ranks that a later turn recovers, where a filter would make the target unreachable.

Rare Phrases

Specific phrases found in the catalog can promote candidates after the base rank is computed. A phrase is worth the inverse of how many products carry it:

Evidence(d) = Σ { 1 / df(p) : p ∈ phrases(d) ∩ phrases(constraints) }

A phrase held by one product is worth 1.0; one held by a thousand is worth 0.001, which is why a bare material word cannot move a ranking. The top 20 of the pool are re-sorted by Evidence with a stable sort, so a product no constraint names keeps its blend position and a turn with no phrase evidence is returned untouched. It reorders already-valid candidates only; an unseen phrase cannot inject an unrelated product. The window is bounded because past it the blend is the only thing keeping a globally rare phrase from pulling an out-of-bucket product up, and the surface is flat from 10 to 40, so the width is not a fitted constant.

This step runs before exposure decides how wide to go, which matters: once the slate narrows to a single product, reordering the served list can no longer change anything. Moving phrase promotion ahead of the exposure decision was worth 0.9633 → 0.9672.

Dynamic Product Exposure

Why a short slate is the right product behavior

Filling all ten slots looks generous and is the obvious default, so it is worth stating plainly why SATU does not do it. The argument does not depend on the scoring formula.

Exposure is irreversible. A product the shopper has seen and scrolled past has been judged. Offering it again later reads as the system not listening. A ten-wide slate on turn one spends ten candidates on the turn when the system knows the least about what is wanted.
A recommendation is a claim, and width is how confident the claim sounds. One product says "I think this is it." Ten products say "I don't know, you look." Both are legitimate things to say; saying the second one every turn trains the shopper to ignore the ordering, and at that point the product is a search box with extra steps.
Attention is the scarce resource, not screen space. A correct item in position seven is not experienced as a recommendation.
Withholding preserves optionality. A plausible product that was not shown is still available next turn, and can then be ranked against the answer that just arrived. A product that was shown cannot be re-offered with the same credibility.
The alternative is worse than it looks. Showing ten every turn means the system never has to commit, so it never has to be right, and the shopper carries all the filtering work the system was supposed to do.

The counterweight is that withholding must be bounded, or confidence becomes stubbornness: a system that keeps insisting on one product while the shopper keeps saying no is worse than one that opens up. Both failure modes are measured at the end of this section.

How many products to show

Three plain rules:

Count how many products are effectively tied for first place.
Show that many.
Unless there is no longer any reason to wait, in which case show ten.

Rule 1: what "tied" means. "Contention" has one exact meaning: the number of top-ranked products whose blended scores S(d) are within a relative margin m of the leader.

C(m) = |{d : S(d) ≥ S(best)·(1−m)}|

The shipped margin is:

m = 0.0005 = 0.05%

Example: if the leader scores 1.2000, a product is still in contention only when its score is at least 1.2000 × 0.9995 = 1.1994.

This is not model fine-tuning and it is not an arbitrary "candidate confidence" label. It is a measured tolerance applied to the deterministic blended scores defined in the previous section.

Rules 2 and 3: the committed head.

head(t) = 10                       if exhausted, or |constraints| ≥ 4, or t > 6
          min(10, max(1, C(m)))    otherwise

The three conditions are the deferral budget. Each states a different reason that holding products back has stopped buying anything: the shopper has nothing left to add (exhaustion, defined under "Exhaustion and Refusal"), enough has already been disclosed to rank on, or six turns have gone by. Without the third, a shopper who never runs dry would be served one product for the whole session, which is the stubbornness failure mode named above.

Finally, products already shown are removed from the pool entirely. If the evaluator continued after they were shown, they are proven non-targets. Before this rule, repeats represented 62.9% of impressions on the hardest test set.

Why 0.05% is the right tolerance

The margin was selected from a sweep rather than from the single best-looking point. Every value from 0 through 0.0009 returns the identical TechnicalScore; performance first falls at 0.001. We chose 0.0005 because it lies in the middle of that stable plateau rather than at its edge. The sweep below was re-run against the shipped agent, so it is directly comparable to the ablation table in the next section.

Relative margin	Meaning	TechnicalScore (delta vs 0.05%)
`0`	0%	0.9672 (+0.0000)
`0.0001`	0.01%	0.9672 (+0.0000)
`0.0005`	0.05%	0.9672 (shipped)
`0.0009`	0.09%	0.9672 (+0.0000)
`0.001`	0.10%	0.9666 (−0.0007)
`0.01`	1.00%	0.9641 (−0.0031)
`0.02`	2.00%	0.9598 (−0.0074)
`0.05`	5.00%	0.9441 (−0.0231)
`0.25`	25.00%	0.8984 (−0.0688)

What that number means in shop terms. At 0.05%, two products count as tied only when their evidence agrees to within five hundredths of one percent. In a real catalog that happens for genuine near-duplicates: the same garment listed by two sellers, or two colorways of one style. Laying both of those out is what a salesperson would do too. Anything merely similar falls outside the band and is held back for the next turn.

Widening the margin does not make the agent more careful, it makes it less honest. At 5% the agent calls a product "tied for first" when it is 5% behind on the evidence, which describes most of any ranked page. The measured head widths show exactly that: at 0.05% the public run produced head widths {1: 415, 2: 1, 10: 37} across 453 turns, while at 5% the width spread to {1: 224, 2: 95, 3: 35, 4: 20, 5: 3, 6: 2, 9: 1, 10: 26}. So the tight margin is not a confidence knob turned up until the score peaked. The flat plateau from 0 to 0.09% means every value in that range returns the same answers. It is a statement that the blended score is a real ordering, and that small differences in it are signal rather than noise.

The distribution also answers the obvious objection. The agent is not being forced to one product by a hardcoded constant: on 415 of 453 turns the ranking genuinely separates, and the 37 ten-wide turns are the deferral budget expiring, which is the system deciding it has waited long enough.

Exposure Ablation

The two degenerate strategies are the honest comparison: always commit to one, or always show ten. Both were measured end-to-end against the shipped rule, after phrase promotion was added.

Strategy	Relative margin	Score	HitRate@10	MRR	MTTC	Avg. shown
Shipped dynamic slate	`0.0005` = 0.05%	0.9672	1.000	0.9750	2.27	1.74
Wider contention	`0.01` = 1%	0.9641	1.000	0.9617	2.22	1.77
Wider contention	`0.05` = 5%	0.9441	1.000	0.8825	2.03	2.20
Wider contention	`0.25` = 25%	0.8984	1.000	0.7045	1.65	6.53
Always show one	Not applicable	0.9638	0.990	0.9900	2.41	1.00
Always show ten	Not applicable	0.8946	1.000	0.6901	1.62	9.84

The two static strategies fail in opposite directions, and each one fails the way the product argument predicts:

Always show ten reaches the target earliest (MTTC 1.62) and then buries it. MRR collapses from 0.975 to 0.690, which means the right product was usually on the page and usually not at the top. That is the "search results, not a recommendation" failure.
Always show one ranks beautifully when it is right (MRR 0.990) and never recovers when it is wrong. HitRate@10 drops from 1.000 to 0.990: two sessions never converge at all, because the agent never opened up. That is the stubbornness failure.

The shipped rule chooses between those two behaviors per turn, and beats both. It gives up a small amount of MTTC against always-ten to gain substantially more MRR, and it gives up a small amount of MRR against always-one to keep full coverage.

Dynamic Question Control

Exposure decides what to show. This stage decides what, if anything, to ask alongside it.

SATU does not use a fixed questionnaire or fixed number of questions. It emits at most one clarification question per turn and may emit none. Nine attribute arms compete:

material, color, size, style, use case,
feature, budget, brand, category

Question options are catalog-derived and limited to two or three plausible values, so every option offered is one that actually narrows the pool.

Why not just ask "anything else?" every turn

There is a tenth arm, a wildcard: "is there anything else you're looking for?". By construction its match set is the union of all nine specific arms, so its expected yield can never be lower. Measured across the public set and the 23 frozen session sets, always asking the wildcard is worth +0.0020 over asking specific attributes. It is the score-maximizing choice and we did not ship it.

The reason is that "anything else?" is not a question any shopper asks or any salesperson asks twice. With specific arms disabled, the agent asks "anything else?" 11,332 times across the evaluation and asks nothing else, ever. That is a system that has outsourced the entire job of knowing what matters about a product back to the customer. With them enabled, roughly three questions in four name a real attribute ("leather or canvas?", "for work or travel?"), which is what makes the conversation feel like a conversation rather than an empty prompt repeated.

The wildcard is kept as a fallback, and it is used when no specific attribute is competitive, which is the case where "anything else?" is genuinely the honest question. The score cost is recorded rather than hidden: specific arms lose on 10 of the 24 evaluation sets and win on 13.

Question Information Value

Both terms of the score read the same weighted structure. Each product contributes its first eight lead lines, feature bullets first and then detail fields, each typed to an attribute and discounted by its position:

offers(d) = { (a, v, wⱼ) : line j of d types to attribute a with value v },
            j = 0 … 7,   wⱼ = 0.35ʲ

The decay encodes that what a listing leads with describes the product while line eight is usually boilerplate; neither term below is a count. Let P be the live pool, capped at the 150 highest-ranked candidates from the ranking stage, and said the values the shopper has already disclosed. Those are excluded throughout, since an attribute they have answered can no longer separate anything.

For attribute a, SATU measures the distribution of its remaining catalog values with Shannon entropy:

weight(v,a) = Σ_{d∈P} Σ { w : (a, v, w) ∈ offers(d), v ∉ said }

p(v|a) = weight(v,a) / Σᵤ weight(u,a)
H(a)   = −Σᵥ p(v|a)·log₂ p(v|a)

Coverage is the complementary term: not how finely the attribute splits the pool, but whether the pool can answer it at all. Each product votes once per attribute, with its strongest line rather than the sum, so a listing naming three materials does not count three times:

best(a,d)   = max { w : (a, v, w) ∈ offers(d), v ∉ said }     (0 if none)

Coverage(a) = (1/|P|) · Σ_{d∈P} best(a,d)
Union       = (1/|P|) · Σ_{d∈P} max_a best(a,d)

Union is the same quantity for the wildcard question, whose match set is the union over every attribute. This is the formal statement of the domination argument made above.

The question score is:

Q(a) = Coverage(a) · H(a) · 0.35^heard(a)

The two terms disagree in practice: entropy asks would this split cleanly, coverage asks can the shelf answer at all, and an attribute holding the highest entropy in the pool still loses if almost nothing declares it. The decay discounts a dimension already described. Refused, remembered-refused, and temporarily avoided attributes score nothing; those come from the refusal state defined in "Conversation State".

SATU then asks the winning attribute only if it is competitive with the wildcard:

ask a* = argmax Q(a)   if  Coverage(a*) / Union ≥ 0.2
         "anything else?"  otherwise

When SATU Stops Asking

The question count emerges from the conversation. SATU suppresses a question when:

the shopper has explicitly run out of preferences
every useful attribute has been refused or exhausted
the final protocol turn leaves no future slate that could use the answer
the behavior switches from information gathering to coverage

After two answered questions add no new constraint, stagnation handling forces the next probe onto an untouched dimension. This prevents SATU from repeatedly asking sharper versions of the same unproductive question.

This makes question selection dynamic in both dimensions: what to ask and whether to ask at all.

Dynamic Behavior Selection

The three stages above (retrieval, exposure, questions) do not each decide independently. One behavior is named per turn and the rest of the pipeline reads it.

Six behaviors compete on every turn:

Behavior	Trigger	Effect
Discovery	Sparse constraints or a broad pool	Seek a high-value preference
Precision	Several decisive constraints	Narrow and commit carefully
Recovery	Shopper pivot or disproven ordering	Replace stale state and rerank
Boundary	Shopper refuses a dimension	Respect the refusal and redirect
Stagnation	Repeated answers add no information	Change the question dimension
Coverage	Few useful turns remain	Stop over-narrowing and expose more candidates

The behavior layer coordinates retrieval, slate shaping, question selection, and response wording. The same shopper can move through several behaviors within one session.

Interpreting Buying and Browsing

We did not assume that "buying" and "browsing" require unrelated retrieval systems. The public sessions show a clear difference in when information arrives, but not in the information ultimately available:

Intent	Constraints before turn 1	Turn 2	Turn 3	Turn 4
Buying	1.00	3.00	4.00	4.00
Browsing	0.00	2.00	4.00	4.00

Buying sessions begin one constraint ahead; both intents contain the same mean number of constraints from turn 3 onward. We therefore interpret intent as a conversation-behavior signal, meaning how quickly SATU should move from discovery to precision, not as evidence that the catalog needs a different semantic retriever. Five attempts to give each intent separate retrieval weights were neutral or negative, so the shipped retrieval constants remain shared.

The public score supports that interpretation: buying reaches conversion in 1.79 turns and browsing in 2.05, while both achieve 100% HitRate@10 and nearly identical MRR (0.967 versus 0.970). The small timing gap follows the one-turn information lead; it is not evidence of weaker retrieval for browsing.

Ranking Recovery

Recovery is the one behavior that fires on proof rather than on a heuristic, so it is worth stating what the proof is.

SATU treats served failures as evidence. An ordering is judged by how much of its own head has already been served and survived:

Spent(o) = |{d ∈ top 40 of ordering o : d already shown}| / 40

The "already shown" set is the same one exposure uses to suppress repeats, so the two mechanisms read one fact. When at least one product has been disproven and Spent(blend) ≥ 0.5, SATU reranks the same candidate pool under an alternative ordering. Half the ranking having been spent without conversion is a fact about that ordering, not a guess about confidence.

We rejected score flatness, read confidence, ranking contention, and raw constraint count as primary triggers because they describe uncertainty but do not prove that the ranking is wrong. A shown product that fails to convert does. This distinction is also the basis of the proposed LLM gate under "Next Steps".

Recovery activates on 2.1% of public turns and 46.4% of turns in the hardest set.

Decision Readiness

SATU computes a bounded turn-level readiness value:

Dₜ = clip(0.7·Currentₜ + 0.3·Dₜ₋₁, 0, 1)

Currentₜ increases with a recognized category, new typed constraints, decisive attributes, urgency, a small candidate pool, and a current pivot. It decreases with a large pool, many previous contenders, browsing without a new constraint, refusals, idle turns, and exhaustion.

The 0.7/0.3 split makes new evidence dominant while retaining some conversational continuity. A decisive correction can move readiness quickly, while the 0.3 prior prevents one vague turn from erasing the session's accumulated direction. Values at or above 0.7 are reported as precision-ready; values below 0.3 are discovery-leaning.

For accuracy, readiness is currently a trace and explanation signal, not an active ranking weight. Its steering switch ships off because the full sweep changed zero behavior decisions: readiness was correlated with evidence already present in the discovery and precision scores. We retain the value because it explains how decision state evolves without claiming an evaluation gain it did not produce.

Evaluation

Scenario	n	HitRate@10	MRR	MTTC	Score
Buying	80	1.000	0.967	1.79	0.9744
Browsing	80	1.000	0.970	2.05	0.9701
Intent override	30	1.000	1.000	4.03	0.9393
Boundary	10	1.000	1.000	2.50	0.9700
Overall	200	1.000	0.975	2.27	0.9672
BM25 baseline	200	0.125	0.068	9.81	0.1067

SATU achieves 9.1× the baseline TechnicalScore and 3.9× fewer turns, with zero exceptions, discarded outputs, or dropped slots.

Test Strategy

The 200 labelled public sessions are close to saturated: 176 of them already convert at rank 1. That leaves 24 sessions of upside against 176 of downside, which is not enough resolution to tell a good idea from a bad one. We therefore use the public set as a regression gate only, and built our own evaluation data on top of it.

The 23 additional session sets are ours, not the organizer's. Each is generated deterministically from the frozen catalog and driven through the organizer's real evaluate() function, so a score taken on them is comparable to a score taken on the public set. They span 99% rank-1 down to 16% rank-1, which is the difficulty range the public set does not cover. The sets vary:

product-description thinness
target popularity
constraint accumulation
shopper pivots
silence and unhelpful answers
target sampling distribution
returning shoppers across multiple visits

Seeds were frozen before any individual feature was evaluated, and are read-only. A set reshaped because of what it showed about a feature would be worthless as evidence. Every measured-off component in the next section was re-swept across all 23 sets, not just the public 200.

Alongside these, 566 automated unit and integration tests cover the agent (455), the measurement harness (108), and the evaluator contract (3).

Gate	Result
Held-out split	Dev 0.9637; held-out 0.9707
Size-biased targets	0.9644
Square-root targets	0.9345
Uniform targets	0.9139
Paraphrased sessions	0.9318–0.9590
Template matching disabled	0.9660
Returning-shopper memory	0.28 fewer turns

Measured Trade-Offs

Features remain disabled when they lose under the frozen evaluation gates, even if they make the architecture appear more sophisticated.

Experiment	Measurement	Shipped state
Per-route popularity weights	+0.014 dev, −0.002 held-out	Disabled
Restart turn budget after pivot	Improved one condition, regressed four gates	Disabled
MMR diversity	Lost on 13 of 18 hard sets	Disabled
Early convergence	MRR fell from 0.909 to 0.781	Disabled
Profile-weighted ranking	Negative on 18 of 18 sets	Disabled
Wider browsing slates	MRR fell from 0.9704 to 0.8942	Disabled
Dense retrieval	Lost on 14 of 15 readable sets	Weight 0

Each component retains its implementation, switch, measurement, and a test confirming its default state.

LLM Reranking (Rejected)

An optional claude-haiku-4-5 reranking tier is fully implemented behind the reranking seam shown in "Retrieval and Ranking", and was tested through 323 live API calls.

Measure	LLM tier	Offline ranker
Score	0.9333	0.9554
Latency per turn	1,087 ms	2.5 ms
Cost per run	$0.385	$0.00

It rescued 6 of 20 difficult sessions but changed enough already-correct rankings to lower the aggregate score. The tier remains implemented as a safe optional layer; every failure path returns the original slate unchanged. It is disabled for scoring, and "Next Steps" describes the gate that would make it worth switching on.

Dense Retrieval (Rejected)

We built a 64-dimensional latent representation of the catalog. It lost on 14 of 15 readable test sets because its strongest dimensions mostly encoded category, which the hard category filter described earlier already represents more precisely, at recall 0.990 against the dense retriever's 0.395. The asset is 4.92 MB, ships at weight zero, and can be deleted without changing scored output.

Stack and Data

Area	Details
Scored runtime	Python standard library only
Core modules	`math`, `array`, `struct`, `re`, `dataclasses`, `pathlib`, `collections`, `enum`, `json`
Tokenization	Custom regex word tokenizer + 31-word stopword list; no learned or subword tokenizer
Development	VS Code, Claude Code, Git, GitHub, GNU Make
Tests	Python `unittest`, 566 automated tests
Runtime APIs	None on the scored path
Optional APIs (off)	Anthropic Messages API, `claude-haiku-4-5`, for the rejected reranking tier
External training data	None
Product catalog	50,000 frozen organizer products
Public evaluation	200 labelled development sessions, supplied by the organizer
Additional evaluation	23 deterministic session sets we generated from the catalog, spanning 99%–16% rank-1

The product catalog is derived from Amazon Reviews 2023 (Clothing, Shoes and Jewelry) by McAuley Lab, UCSD, and was verified against the published SHA256 checksum. SATU uses no scraping, catalog mutation, or manually labelled training data.

NumPy is used only by a standalone offline preprocessing script that builds the disabled dense asset. It never runs at scoring time and is never imported by the scored agent.

Limitations

Each limitation below names what it costs, and the "Next Steps" item that addresses it follows in the same order.

Constraint language must map reasonably well to attributes found in the catalog. SATU's understanding of a request is lexical and catalog-derived. Implication: a shopper who describes a product in vocabulary the catalog never uses, such as a use case, an occasion or a vibe, gets ranked mostly by the popularity prior, which is a good fallback but not an answer to what they said.
Information-value scoring estimates product separation, not an individual shopper's willingness to answer. Q(a) measures what the catalog can distinguish; disclosure depends on how the shopper thinks about their own preference. Implication: SATU can ask the theoretically most informative question and get "I don't know" back, spending a turn on a dimension the shopper was never going to be able to specify.
The evaluation interface gives no way to tell which sessions belong to the same shopper. The published contract closes both the reset request and the user profile to additional fields, and the evaluator issues a fresh session id every time, so there is no channel an identity could travel in. We had to assume every session is a different person. Implication: cross-session memory is implemented and observable only through our own harness, which supplies identity beside the real evaluator; the reported 0.28-turn saving comes from 66 synthetic shoppers over 3 visits each, which is a demonstration rather than a validation.
Decision readiness resets between shopping missions to avoid transferring stale confidence. Implication: a returning shopper who is genuinely mid-decision restarts at discovery-leaning readiness, so the first turn of a second visit is less decisive than it could be.
The exposure gain is conditional on how hard the private sessions are. Withholding slate slots is worth +0.0160 on the public 200 and every pessimistic reweighting agrees, but on our hardest frozen sets it reads +0.0092 in favour of showing more, because there coverage binds rather than rank. Implication: if the private set is substantially harder than the public one, the single largest design decision in the system is pointed the wrong way.

Next Steps

Reach beyond catalog vocabulary for constraint understanding, so a request phrased as a use case rather than an attribute still contributes ranking evidence instead of falling through to the popularity prior.
Model shopper-specific question answerability alongside product information gain. Today we approximate this by treating every session as a distinct shopper with no answering history, so Q(a) is the only term available. Given a shopper who returns, the same score can be weighted by which dimensions that person has actually been able to answer before.
Evaluate cross-session memory under large-scale testing with a real identity channel. The mechanism exists; what is missing is a population large enough, and real enough, to say whether the saving holds outside our 66 synthetic shoppers.
Learn when decision readiness can safely persist across visits, rather than resetting it unconditionally, so a returning mid-decision shopper does not restart from discovery.
Make the short slate coverage-conditional. It is the one shipped decision whose sign depends on the difficulty of the evaluation set, so gating it on a coverage estimate rather than shipping it unconditionally is cheap insurance against the last limitation above.
Gate LLM reranking on expected benefit rather than enabling it globally. The tier lost as measured because it was applied to every turn, including the ~90% where the offline ranker already puts the target at rank 1. Its errors on those outnumbered its rescues by roughly 22 to 1. A gate fixes the exposure, not the model: invoke it only on turns where the deterministic ordering has been proven wrong, which is the same Spent(o) signal Ranking Recovery already computes. On the public run that is 2.1% of turns, so the tier would cost about $0.01 per run and roughly 23 ms of average per-turn latency, while touching only the turns the offline path has already failed.
Evaluate the in-memory architecture at catalog sizes approaching five million products.

Team Contributions

Member	Area	Contribution
Justin Stevenson Theodorus	Retrieval, ranking, and measurement	Retrieval and ranking, and the measurement harness the whole project is argued from: the robustness gates, the synthetic session sets, and the sweep that re-reads every component where it still has room to move
Angelica Gonathan	Deferred commitment and slate exposure	Deciding how much of the ranking a turn should reveal, and deriving that from how well the ranking separates rather than fixing it in advance. The largest scoring gain in the project
Catherine Kang	Dialogue and response layer	The customer-facing side of the dialogue: what the agent says about the slate it is serving, how it reads a constraint back, and what asking a real question costs
Azka Tazkiatunnafsi	Demo and narrative	The demo: its flow and narrative structure, and the video itself