Fit Check

Record a short 360° outfit video in the browser. The backend runs it through TRIBEv2 video-only inference and returns a short mp4 of your predicted brain activity rendered on the fsaverage5 cortical surface.

• Frontend (/frontend): Next.js 16, React 19, Bun. Single client page at app/page.tsx — camera preview via getUserMedia, MediaRecorder for capture, fetch POST to the backend, <video> tag for the result.
• Backend (/backend): FastAPI + uv. Loads TribeModel once at startup with features_to_use=["video"] so no text (Llama) or audio (Wav2Vec) weights are ever fetched or loaded. POST /analyze accepts a multipart video, runs V-JEPA2 + DINOv2 feature extraction + the TRIBEv2 fMRI head, renders frames via PlotBrainNilearn.plot_surf, and stitches an mp4 with ffmpeg (bundled via imageio-ffmpeg).

• NVIDIA GPU with CUDA (tested on RTX 3090, 24GB VRAM)
• Disk: the TRIBEv2 checkpoint + V-JEPA2 + DINOv2 weights total ~7GB. If your workspace has a tight quota, point HF_HOME at a location with more room (see below).

cd backend
UV_CACHE_DIR=/tmp/uv-cache UV_LINK_MODE=copy uv sync
HF_HOME=/tmp/hf-cache .venv/bin/uvicorn main:app --host 127.0.0.1 --port 8000

First request takes ~1 minute on short clips (V-JEPA2 encoding is the bottleneck). The model is loaded once on startup and reused across requests.

cd frontend
bun install
bun run dev

Open http://localhost:3000, grant camera permission, tap Record, spin once slowly, tap Stop. After a ~1 minute wait the brain video appears.

• from_pretrained(..., config_update={"data.features_to_use": ["video"]}) is the official config knob. It stops the data loader from instantiating the text and audio feature extractors, which means Llama-3.2 (gated) and Wav2Vec-BERT never load. No HF_TOKEN needed.
• remove_empty_segments = False is set on the model so segments without Word events (which is every segment here, since the audio/text path is disabled) are kept in the prediction output.
• Events df is hand-built with a single type="Video" row. The upstream get_audio_and_text_events helper is skipped entirely — it would try to extract audio and run WhisperX.
• Plotting uses PlotBrainNilearn (matplotlib-based) instead of the default PlotBrainPyvista because the container is headless (no X / EGL / OSMesa).
• TRIBEv2's plot_timesteps_mp4 calls plot_surf(axes=[plain_2d_ax]), which is incompatible with the Nilearn backend (needs 3D-projection axes). We drive the render loop manually: get_fig_axes → plot_surf(axes=ax) → savefig → ffmpeg.