ShadowSAT | Devpost

system_architecture
real_student_agent_loop
counterfactual_policy_engine
production_vs_evaluation_boundary

Inspiration

ShadowSAT started with a simple observation: two students can know the exact same math, yet one of them can still fall apart when the clock starts. Most SAT tools look at the final answer. Right or wrong. Weak algebra. Strong reading. More practice here. Less practice there. But that did not feel like the whole story. A student might understand algebra and still lose points because they spend too long on one question, second guess correct answers, avoid using the calculator when it would save time, or keep pushing through instead of flagging and returning later. A good tutor notices those patterns. Many students in underserved schools do not have access to that kind of personal observation. That became the idea behind ShadowSAT. Don't just take the SAT. Send your Shadow first.

Instead of only asking what a student knows, ShadowSAT tries to understand how that student behaves during a timed test.

What it does

ShadowSAT builds a behavioral model from a student's real practice attempts. While the student works through SAT-style questions, the system records signals such as response time, answer changes, confidence, flags, revisits, calculator usage, question position, and remaining time. From that evidence, ShadowSAT builds a probabilistic behavioral twin. It might discover that a student understands linear equations well, but regularly spends too long on difficult word problems. It may find that they tend to change correct answers during review, or that their accuracy drops near the end of a timed module. ShadowSAT then asks a different question: What would happen if this student took the same kind of test using a different strategy?

It generates interpretable test policies and simulates thousands of possible future attempts using the student's behavioral model. For example: flag difficult questions after 60 seconds reserve four minutes for review use the calculator first on certain eligible problem types avoid reopening high-confidence answers switch to a two-pass strategy when time pressure becomes high The system compares these strategies under uncertainty and selects the most robust one. It does not just display advice. It creates a decision receipt explaining what it considered, what it selected, why it selected it, and how confident the system is. The student then takes another real timed module using that policy. ShadowSAT watches what actually happens and decides whether to keep, revise, or revoke the strategy. That closes the loop.

How we built it

We designed ShadowSAT around one rule from the beginning: The intelligence should live in the engine, not in a chatbot.

The core system is deterministic and local-first. Real student interactions flow into an append-only telemetry ledger. The system reconstructs each practice session and independently calculates correctness from trusted question content rather than trusting the browser to report whether an answer was right. From that event stream, the engine extracts behavioral features and builds an uncertainty-aware student model. Instead of pretending that a few questions are enough to perfectly understand someone, ShadowSAT keeps track of how much evidence it actually has. When the evidence is too weak, the system can refuse to change the student's strategy and ask for more observations. Once enough evidence exists, the counterfactual engine samples many plausible versions of the student from the model's uncertainty and runs candidate test policies against them. The policy optimizer considers more than average performance. It also looks at completion probability, unanswered questions, timing stability, lower-tail outcomes, and strategy complexity. That means ShadowSAT can intentionally choose a slightly lower average strategy if it is much more reliable under pressure. The backend is designed to work without an LLM or constant internet connection. The same core logic can run locally in the browser, while the frontend stays intentionally small and mainly acts as an instrument for capturing real behavior and showing the engine's decisions. We also separated production behavior from evaluation. Synthetic personas are used only to test whether the engine can recover known behavioral patterns and choose policies close to an oracle. They never power a real student's model.

Challenges we ran into

The hardest problem was making sure ShadowSAT was genuinely intelligent without pretending to know more than the data could support. It is easy to build something that says: "You are bad at pacing."

It is much harder to build something that can explain exactly which observations support that conclusion, how uncertain the conclusion is, and whether the evidence is strong enough to justify changing the student's strategy. We also had to avoid circular evaluation. If the same simulation model creates a student and then evaluates itself on that student, the results can look impressive without proving much. We solved this by separating the hidden evaluation model from the model ShadowSAT actually learns. Another challenge was keeping the system realistic. A simulation can run 10,000 futures, but those futures are only useful if the student model underneath them came from real behavior. That pushed us to make real attempts, real telemetry, real longitudinal learner state, and real retesting part of the production path rather than relying on a prebuilt demo persona. We also had to be careful with our claims. ShadowSAT does not predict an official SAT score and does not claim that a strategy causes score improvement. It reports what happened in practice and what the current behavioral model predicts. That made the project more credible, even if it meant avoiding some flashy claims. Accomplishments that we're proud of The part we are most proud of is that ShadowSAT is not just a recommendation screen sitting on top of an LLM. The core decision actually comes from the system. A student can complete a real timed attempt, generate real behavioral telemetry, build a learner model, run thousands of counterfactual simulations, receive a selected policy, complete another real attempt, and have the system update its decision based on what happened. We are also proud of the transparency. Every major decision can be traced through: student action → recorded evidence → behavioral estimate → uncertainty → simulated strategies → selected policy → decision receipt → retest We built the engine so that it can also say: "I do not have enough evidence yet."

For us, that is an important feature rather than a failure. We also designed ShadowSAT to remain useful on limited hardware and unreliable internet connections. The core intelligence does not require a cloud model call every time the student answers a question. Perhaps the part that best captures the project is still the simplest idea: Before the student tests a strategy on the real clock, their Shadow can test thousands of possibilities first.

What we learned

The biggest thing we learned is that personalization is not the same as adaptation. Giving two students different question sets is personalization. Watching how one student behaves, forming a hypothesis, choosing between multiple actions, testing that decision, observing the result, and changing future behavior is much closer to real agency. We also learned that uncertainty matters a lot in education software. Students do not generate perfect datasets. Their behavior changes from day to day. A single slow question does not mean someone has a pacing problem. Building around uncertainty forced us to make better decisions and helped prevent the system from confidently overreacting to noise. Another lesson was that simpler models can sometimes make a stronger product. Instead of hiding everything inside a giant black box, we wanted judges, students, and developers to be able to understand why ShadowSAT made a decision. That pushed us toward interpretable behavioral models, explicit policy rules, reproducible simulations, and inspectable receipts.

What's next for ShadowSAT

The next step is to move from strong engineering validation to larger real-student evaluation. We want students to use ShadowSAT over multiple practice sessions so we can study how stable the behavioral models are, which policies consistently survive real retesting, and when the system should choose to gather more evidence instead of acting. We also want to expand the original SAT-style question bank while keeping the core experience lightweight and accessible. Longer term, the most exciting direction is bigger than the SAT. The same idea could apply anywhere performance depends not only on what someone knows, but also on how they behave under constraints. Certification exams, technical interviews, timed assessments, and other high-pressure learning environments all have the same hidden layer. For now, ShadowSAT focuses on one question: You have studied the material. But when the clock starts, are you actually using the strategy that works best for you?

ShadowSAT sends your Shadow ahead to find out.