Inspiration
I casually asked a surgeon: what's the one tool you wish you had?
The answer wasn't a scheduling app or a note-taker. It was this - the night before a case, or five minutes before scrubbing in, they want to see the specific anatomy they're about to operate on. Not a textbook plate drawn for the average human. Not a YouTube video of someone else's patient. The actual spatial relationships for this approach, this side, this variation, in a view they can rotate and interrogate.
What exists today is either static (atlases, illustrations) or generic (pre-built 3D models that show you one canonical body and nothing else). Nothing takes a sentence like "right hepatic hilum for a lap chole, show me the hepatocystic triangle and the structures at risk" and builds you that scene.
So I built whatever the response I got. This is the result.
What it does
SeeIn turns a surgeon's prompt into an interactive 3D operative scene and iterates until the anatomy has been verified.
You describe the case in plain language. SeeIn asks the clinically meaningful clarifying questions first: anatomy, laterality, approach, anatomical variation, audience, teaching goal. Then it runs three grounded medical research branches in parallel plus a reference-image search, and synthesizes an evidence dossier.. object studies, critical spatial relationships, contradictions between sources, and readiness gaps.
Nothing gets generated until you approve that dossier. That gate is deliberate. If a surgeon is going to look at a generated organ, they should first see what it was built from.
After approval, the system pulls stable anatomical registrations from a static atlas, asks Gemini for complete request-specific React Three Fiber source, validates and compiles it, renders every required camera view with real Chromium via Playwright, and sends the actual PNG back to Gemini for multimodal inspection against the approved intent topology, scale, laterality, tissue planes, operative corridor, critical relationships.
If a view fails, the whole module gets replaced and the loop runs again. All views must pass at the same source revision. Only then is it accepted, and its placement metadata gets promoted into a reusable anatomy library.
There is no fallback path. No placeholder PNG, no fixture AI, no "here's a generic primitive scene instead." Failures surface as failures.
How we built it
We first sat down and figured out what the architecture would look like before writing product code. That turned out to be the highest-leverage hour of the project.
Then we built multiple scenes in Three.js by hand. This was the part that felt like a detour and wasn't. Hand-building real surgical scenes is what taught us what the machine would need to be told that a gallbladder isn't a shape, it's a shape at a position in a frame relative to two named landmarks at a specific scale. You can't learn that from the outside.
That work became reference-projects/surgical-atlas: a 175 cm CC0 MakeHuman base at 13.4 atlas units, supine patient, and registered operating-room, abdominal, thoracic, pelvic, and right-groin frames with organ centres, nominal sizes, axes, and camera starts.
Then we handed the whole thing.. architecture, the pros and cons we'd already argued through, the reference scenes.. to multiple coding agents and had them figure out the rest. Having the tradeoffs already written down mattered more than the code did. The agents didn't have to guess what we'd rejected.
The runtime that came out of it:
- Fastify + TypeScript backend owning research, placement retrieval, generation, validation, compilation, rendering, inspection, and acceptance. The frontend only draws the graph and mounts a compiled module.
- A resumable agent graph that checkpoints at every node, so a run that dies at hour two doesn't restart from the prompt.
- A Zod placement contract every generated structure must satisfy —
registered(exact frame, centre, size, rotation from the atlas) orresearch-derived(anchored to at least two named landmarks). - A source sandbox. Generated modules may import React, Three.js, and
@seein/atlas— nothing else. Network calls, storage, dynamic imports, and Node APIs are rejected before anything compiles. - esbuild +
tscon model output, then Playwright for real renders, then Gemini multimodal QA on the real pixels. - ClickHouse for run telemetry, content-addressed caching on research, synthesis, and renders.
Challenges we ran into
Anchoring internal organs in 3D space. This was the hard problem, and it stayed the hard problem. A model that can describe the hepatocystic triangle perfectly in prose will place the gallbladder floating six inches outside the abdomen, buried inside the liver, mirrored to the left side, or scaled like a grapefruit. Every time. The text is right and the geometry is nonsense.
The fix was to stop asking the model for positions at all where we could avoid it. Registration got separated from morphology: the model generates new visual anatomy from the approved research every time, but the registry owns where things live. Known structures that drift from their registered centre or nominal size get rejected. Anything that escapes its frame bounds gets rejected.. the validator literally checks |centre| + size/2 against the frame on every axis. New structures have to declare at least two registered anchors.
That doesn't make the model place things correctly. It makes wrong placements impossible to accept, which turned out to be the achievable version.
Making QA mean something. Early on it was tempting to have the model check its own source code. That catches nothing.. the code always looks right. The only honest check was rendering real pixels in real Chromium and making the model look at the image. Which forced an uncomfortable rule: a source correction invalidates every prior passing view, because all views must prove the same revision. One fix at the end means re-rendering everything. Slow, but a scene where four views passed at revision 2 and one passed at revision 5 isn't a scene that ever existed.
Everything is a moving target at once. When a run fails you're debugging research quality, generated code, placement math, the compiler, the renderer, and the QA judge simultaneously. Our verification run on the hepatic hilum reached a passing QA verdict at revision 5 and then died on a TypeScript error.. the model had put clearcoat on a meshStandardMaterial. Correct anatomy, wrong material prop, whole run lost. Strictness cuts both ways, and we kept it anyway.
Cost and runaway loops. A system that regenerates until it's right will happily regenerate forever. We had to bound it: max runtime, max logical AI calls, max source repairs, max QA targets, max research rounds. And the accepted library retrieves small placement records, never prior source — otherwise every new prompt slowly becomes a copy of the last scene.
Accomplishments that we're proud of
It generates scenes far better than we expected. Going in, the honest expectation was "recognizable blobs in roughly the right place." What comes out is a real operative scene... our accepted right hepatic hilum module carries the liver visceral surface at segments IVb/V/VI, gallbladder with fundus, body and Hartmann's pouch, cystic duct, main biliary axis, cystic and right hepatic arteries, Rouvière's sulcus, the hepatocystic triangle, the cystic plate, and laparoscopic graspers... each tied back to its own study in the approved dossier.
The loop. Genuinely the thing we're proudest of. You give it a prompt and it will grind for hours..research, generate, compile, render, look, reject, replace, render again... until it converges on something that survives inspection from every required angle. Watching a scene fail QA at revision 2 and come back correct at revision 4 without anyone touching it is the moment the project stopped feeling like a demo.
We never built the escape hatch. At every point where the honest engineering move was "add a fallback so the demo always shows something," we didn't. No fixture AI, no placeholder image, no primitive-scene fallback, no alternate generator. Fixture output cannot satisfy the production contracts... we made that structurally true, not a promise. For a tool a surgeon might one day trust, a plausible-looking wrong answer is worse than an error message.
What we learned
Separating stable structure from generated content is the whole game. The single biggest architectural win was refusing to let the model own both morphology and placement. The registry is deliberately not a shape library... it prevents floating, buried, mirrored, and mis-scaled structures, and nothing else. Every prompt still produces new anatomy. Constrain the axis where models fail, leave free the axis where they're strong.
Verify the artifact, not the intention. Checking generated source tells you the model was confident. Rendering the actual PNG and looking at it tells you whether the anatomy is there. Any QA that doesn't touch the real output is theater.
Full replacement beats patching. Our first instinct was local repairs... nudge a transform, swap a primitive. It produced incoherent Frankenstein scenes where each fix broke a relationship elsewhere. Deleting the module and regenerating from the same approved evidence converges faster and stays internally consistent. So local patches don't exist in the codebase at all.
Human gates belong before generation, not after. We put evidence approval upstream of any 3D work. It's cheaper to correct research than to correct a scene built on bad research, and it means the surgeon reviews claims in their own domain rather than being asked to rubber-stamp geometry.
Sandboxing model output isn't optional. Once you're compiling and executing generated TypeScript, an import allowlist and forbidden-pattern rejection are load-bearing infrastructure, not hardening you add later.
Provenance is a feature. Prior scene source stays as audit evidence and is never copied into an unrelated prompt. Revision feedback opens a linked project so the original evidence and accepted result survive untouched. In a clinical context, being able to answer "where did this come from" is worth as much as the render.
What's next for SeeIn
Reliability first. The generation loop converges, but not consistently enough. Cutting the failure modes.... the compile errors on correct anatomy, the placement retries, the runs that burn their budget before converging... is the whole roadmap right now. Nothing else matters until a surgeon can expect a scene rather than hope for one.
Get it in front of real surgeons. Everything so far has been built from one conversation and a lot of inference. The next version needs to be tested by people who will immediately spot the things we can't... the anatomical detail that's subtly wrong, the view that's technically accurate but useless from the operative perspective.
Voice, and the pre-op moment. The real use case is a surgeon talking to it... asking for the visual they want to look at before they head into the OR or let it take live images and construct live visuals, then asking follow-ups out loud while it's on screen. Typing a paragraph-long prompt is a prototype affordance, not the product.
Custom modules. Let surgeons and institutions build and register their own components... a specific prosthesis, an approach they teach a particular way, a departmental convention... and have the generator reference them directly. The accepted-anatomy library already does this automatically for machine-generated structures; opening it to human-authored modules is the natural next step, and it's what turns SeeIn from a generator into something a department actually owns.
Log in or sign up for Devpost to join the conversation.