Watches you work. Takes over when you hit the wall.

FlowState is an ambient developer wellness agent. It monitors your facial expressions via webcam in real-time, detects stress and fatigue using Gemini 2.0 Flash vision, and when you cross the threshold — with your consent — Claude Computer Use takes over your GCP Cloud Shell to fix the problem while you rest.

Webcam (5s tick) → Gemini Flash (sentiment) → State Machine → Consent Modal → Claude Takes Over

1. Webcam capture — OpenCV grabs frames every 5 seconds (rolling buffer of 5)
2. Sentiment analysis — Gemini 2.0 Flash scores each batch for stress + fatigue (0-1)
3. Threshold detection — 3 consecutive high-stress readings (15s) triggers the handoff
4. Consent modal — You approve or dismiss (30s timeout, auto-dismisses)
5. Context gathering — Gemini reads your screen + Augment Code reads your codebase
6. Autonomous takeover — Claude Computer Use opens GCP Cloud Shell and fixes the issue
7. Live feed — Every agent action streams to the UI in real-time via SSE

STREAM A (always on):  webcam → sentiment_agent → score_queue
STREAM B (always on):  screen capture → rolling_buffer

ORCHESTRATOR:          score_queue → state_machine → handoff_trigger

HANDOFF PIPELINE:      negotiator (consent modal)
                       → assessor (Gemini + Augment Code)
                       → control (Claude Computer Use)
                       → done → reset

cd flowstate
cp .env.example .env   # Fill in your API keys
pip install uv
uv sync
uv run flowstate

cd flowstate/ui
npm install
npm run dev

Open http://localhost:3000 — the ambient status bar appears at the top.

// Status updates (continuous)
{"type": "status_update", "data": {"state": "calm|elevated|stressed|critical", "consecutive": 2, "stress": 0.7, "fatigue": 0.6}}

// Handoff trigger (when threshold crossed)
{"type": "handoff_trigger", "data": {"evidence": "...", "task_summary": "..."}}

// Agent actions (during takeover)
{"type": "action", "data": {"step": "Running gcloud...", "icon": "terminal|browser|search|check"}}

// Completion
{"type": "done", "data": {"message": "Task complete. Control returned to you."}}

Built for the Multimodal Frontier Hackathon, March 2026.