Prototype Lens Studio project for capturing teacher demonstrations and replaying them as AR-guided student tutorials on Spectacles.

The Lens Studio project lives in OverShoulder/.

OverShoulderWebBackend/ is a reference snapshot of the web/backend project.

• Home: loads tutorial templates from the backend and lets the user open Teacher or Student flows.
• Teacher: records teacher narration with ASR, captures visual context, generates tutorial drafts, and uploads capture data.
• Student: runs a selected tutorial, scans the camera/depth view, places AR markers, and provides optional OpenAI voice help.

• OverShoulder/Assets/Scripts/StudentVoiceAssistantOpenAI.ts: shared OpenAI Realtime assistant for Home, Teacher, and Student. Home/Teacher mic starts off by default; Student mic starts on. Gemini scan updates are sent as context without forcing speech.
• OverShoulder/Assets/Scripts/StudentHelperGemini.ts: student-side Gemini vision pass. Returns step completion, confidence, situation, hints, suggested step, and marker bounding boxes.
• OverShoulder/Assets/Scripts/GeminiAPI.ts: teacher-side Gemini vision pass. Describes what is visible during teacher capture.
• OverShoulder/Assets/Scripts/TeacherAIController.ts: teacher-side Gemini text pass. Generates tutorial titles, step drafts, completion rules, and marker definitions.
• OverShoulder/Assets/Scripts/ASRController.ts: Snap ASR wrapper for teacher recording transcripts.
• OverShoulder/Assets/Scripts/DepthCache.ts: caches camera/depth frames and projects Gemini image-space marker positions into world space.

• OverShoulder/Assets/Scripts/AppModeController.ts: switches Home, Teacher, and Student modes; loads tutorial lists/bundles.
• OverShoulder/Assets/Scripts/SceneController.ts: owns Teacher capture state, ASR transcript timeline, visual-context scans, AI drafts, and uploads.
• OverShoulder/Assets/Scripts/StudentRuntimeManager.ts: owns Student tutorial runtime, repeated scans, marker placement, step progression, and OpenAI context updates.
• OverShoulder/Assets/Scripts/ResponseUI.ts: renders student AR markers: labels, dots, lines, arrows, hand hints, and rotation markers.

This repository also includes a reference copy in:

OverShoulderWebBackend/

That folder contains a reference snapshot of the web editor, backend API, D1 migrations, and tutorial JSON schema handling. The web/backend remains a separate project; this copy is kept here so the Lens-side behavior and backend contract can be reviewed together.

Backend responsibilities:

• Store teacher capture sessions, events, transcripts, visual context, step drafts, reviewed step JSON, and published tutorial metadata.
• Serve published tutorial summaries to the Home mode.
• Serve one complete tutorial bundle to the Student mode.
• Preserve backend-defined published ordering.
• Normalize marker objects and tutorial bundle shape for the Lens runtime.

The backend does not run Gemini/OpenAI reasoning for the Lens runtime. AI prompting, camera/depth interpretation, marker placement, and voice guidance are owned by Lens scripts.

Published tutorials are represented as one complete JSON bundle. Student Lens downloads the full bundle at launch; it does not fetch step data one step at a time.

Core bundle fields:

• id: stable tutorial id used to fetch the bundle.
• schemaVersion: current value is 1.
• title: title shown on tutorial cards and runtime UI.
• description: short card subtitle.
• runtime.autoJump: whether Lens can auto-advance after repeated completion checks.
• intro: title, goal, and global context.
• targets: reserved for future global task objects.
• steps: ordered runtime steps.

Each step can include:

• stepIndex
• title
• instruction
• detect
• completion
• context
• rawTranscript
• markers

Markers constrain what the student Lens may place in the world. Supported marker types are:

• label
• large_label
• dot
• hands
• line
• arrow
• rotation

Marker ids are assigned by the web editor. Tutorial authors edit marker type, text, location fields, hand hint settings, and rotation direction.

Home tutorial list:

GET /api/tutorials?limit=50

Selected tutorial bundle:

GET /api/tutorials/:id/bundle

Teacher capture lifecycle:

POST /api/capture-sessions
PATCH /api/capture-sessions/:id
POST /api/capture-sessions/:id/events
POST /api/capture-sessions/:id/transcripts
POST /api/capture-sessions/:id/steps

1. Home mode requests GET /api/tutorials?limit=50.
2. The backend returns published tutorial cards in the configured order.
3. The user selects a tutorial.
4. Student mode requests GET /api/tutorials/:id/bundle.
5. StudentRuntimeManager parses the bundle and starts the scan loop.
6. Lens sends active step context and marker location requests to Gemini.
7. Gemini returns completion state, hints, and marker matches.
8. Depth Cache projects marker locations into world space.
9. ResponseUI renders labels, dots, hand hints, lines, arrows, or rotation markers.

This project targets Spectacles and Lens Studio. It depends on Remote Service Gateway, Spectacles Interaction Kit, Spectacles UIKit, ASR, camera, and depth modules.

Generated Lens Studio cache folders and web build artifacts should not be committed.