KWAK | Devpost

Inspiration

Everyone has said "where are my keys?" to an empty room. Smart glasses were supposed to fix this. Google Glass and Spectacles both died the same death: a camera on your face with no brain behind it. Meanwhile Meta Ray-Bans and the Even Realities G2 proved people will actually wear this stuff. We wanted the missing piece: an ambient agent that watches and listens all day, decides on its own when something matters, and hands you the answer before you finish asking. No wake word, no button, no "Hey assistant." Just you, talking like a human, and a system that remembers everything so you don't have to.

What it does

KWAK is an always-on, retroactive glasses assistant. A camera + mic rig (today: a Raspberry Pi 4 with Camera Module 3 streaming over a single USB-C cable; glasses hardware next) feeds a memory system that never stops recording reality: scenes, objects, faces, and every word spoken around you.

Then it acts on it, retroactively and unprompted:

Ask "where are my keys?" hours after you put them down, and it recalls the exact moment with photo evidence.
Meet someone and just say "nice to meet you, Sarah". It enrolls their face from the natural introduction and recognizes them forever after.
Say "tell Kenny about Vitamin B when I see him". It waits, watches, and fires the reminder when Kenny's face shows up.
Every conversation, class, and scene is searchable, correctable, and deletable in a full memory browser.

The magic is the gate: TypeSafe's Jev, a System One model that returns typed decisions (booleans, choices, scores) in ~100 ms instead of generating text. Every few seconds the world (transcript, faces, scene, timing) gets serialized, and Jev answers: is the wearer talking to us? Does this need action now, later, or never? That's how there's no wake word: the model reads the room.

How we built it

Three brains, cleanly separated:

Perception (Python) with InsightFace for face ID, a shared gallery, and a Jev-guarded name-binding flow (an exact spoken name binds to a stable face, nothing else), plus the Whisper API for speech-to-text.
Memory (TypeScript/Node) with timestamped raw sources in SQLite plus hourly JSONL journals, FTS5 literal search, entity/scene memory with evolving attributes, OpenAI vision for scene understanding, and a capture PWA with Live / Memory / Debug views.
Agent (Bun) where Jev routes start/update/cancel decisions into an agent harness: OpenAI-powered turns, reminders, evidence graphs, subagents, code/browser tools, and push notifications.

The device side speaks one tiny wire protocol (binary JPEG/PCM frames + JSON control over a WebSocket) that anything can implement: laptop sim, phone, Pi, ESP32 later. The Pi runs as a systemd service on the Pi 4's hardware MJPEG encoder (720p24, hot-swappable to 1080p from a live ops dashboard with sparkline graphs for fps, bitrate, and frame staleness). The whole stack is mock-first: the flagship demos also run as deterministic scenario tests with zero API keys.

Challenges we ran into

The physical layer tried to kill us, repeatedly. Campus WiFi is 802.1X (unshareable), the hotspot dropped the Pi constantly, and mDNS died on every network. At 5 AM the Pi looked completely dead, so we pulled the SD card and did filesystem forensics on the Mac (e2fsck and debugfs on the raw ext4 partition), proving the Pi had been alive the whole time, faithfully logging Network is unreachable every 5 seconds. The WiFi was the liar.

The fix became our favorite hack: one USB-C cable = power + network. We flipped the Pi into USB gadget mode with two boot-partition edits, discovered NetworkManager silently ignores gadget interfaces, injected the fix into the powered-off SD card with debugfs, then caught the Pi over IPv6 link-local to pin static addresses. Result: 0.6 ms ping, zero drops, forever.

Streaming had its own gremlins. TCP bufferbloat made the feed "age" under load, fixed with UDP-style semantics: two-frame socket buffers so stale frames drop instead of queueing. A phantom staleness ramp turned out to be clock drift between the Pi's crystal and the Mac (~40 ppm), fixed with a sliding-window delay baseline. Also, Python ate a \n inside inline JavaScript and took down the whole dashboard with one invisible character.

Accomplishments that we're proud of

A genuinely wake-word-free ambient loop: natural speech + vision, typed Jev decisions, useful action, with raw memory preserved alongside.
Face enrollment from ordinary introductions, guarded so a quoted or ambiguous name can never corrupt an identity.
A memory with receipts: every claim traces to raw evidence, and corrections/deletions propagate everywhere.
The single-cable Pi rig: 24 fps, 0-2 ms staleness, reboot-proof, tuned like production infrastructure.
Engineering discipline at hackathon speed: frozen contracts let parallel humans and AI agents build in separate lanes without collisions, every external dependency behind a swappable backend, audited and tested end to end.

What we learned

System One models change the architecture. When ten typed questions cost the same 100 ms as one, you stop parsing commands and start asking the model to read the room.
Real-time means dropping, never buffering. Every queue in a live pipeline is latency debt; newest-wins beats reliable-but-stale everywhere.
The bottom of the stack eats the top. Power supplies, cables, DHCP, 802.1X, clock drift: we debugged more physics than Python.
Your laptop can read, repair, and even edit a dead Pi's filesystem offline. debugfs is a lockpick.
Evidence beats vibes: every "it's broken" this weekend had a different, measurable cause.

What's next for KWAK

Real glasses. The wire protocol was designed from day one for an ESP32-S3 frame; the Pi rig is a stand-in, not the product. The hardware roadmap: an integrated mini display so answers, reminders, and profile cards land right in your field of view instead of on a phone, and onboard speakers so KWAK can talk back, confirm a reminder, or whisper a name you forgot without you ever looking down.

On the software side, the big one is world model regeneration and replay: because every raw frame, utterance, and percept is journaled, we can rebuild the entire world model from scratch with better models later, and scrub back through your day like a timeline. Ask a question about last Tuesday and KWAK re-derives the scene, not just a summary of it. Replay also makes debugging and demos deterministic: any moment can be re-run through the full pipeline exactly as it happened.

Beyond that: richer open-vocabulary object tracking for a deeper world model, smarter retroactive tasks, multi-wearer identity done safely, and privacy as a first-class feature with explicit enrollment consent, on-device redaction, and memory you can audit, correct, and truly delete. The goal hasn't moved: a second memory that's always on and never in the way.