SWISHHH! | Devpost

Home Screen
Shot Incoming
Point Gain
Tutorial page
Settings for Game Sync
Statistics

Inspiration

We kept hearing friends argue during NBA watch parties about whether the next shot would go in. Everyone had a gut feeling, but you know YOURS is correct. We wanted an interactive experience that fused real data and live video monitoring so those “I told you so” moments were actually grounded in evidence.

What We Built

A live scoreboard that mirrors the official NBA feed.
An in-game viewer that streams play-by-play plus lightweight box score deltas into slick Tailwind/Framer-motion overlays.
A webcam gesture powered by TensorFlow MoveNet + MediaPipe where you move your arms in a baseketball shot motion to “call” the next make or miss.
API routes that stitch scoreboard, play-by-play, and cached mock data into one normalized payload for the UI and for the Gemini pose-to-text experiments.

How We Built It

Data plumbing: A pair of Next API routes poll the NBA JSON endpoints to get live data and sanitize them - all while the games are still happening. We also get player stats (FG%, 3P%) to weight the risk of predictions.
Front-end shell: The app router and Tailwind theme (with custom gradients from app/globals.css) render the live dashboard, while Framer Motion handles micro-interactions on cards and score widgets.
Gesture intelligence: The webcam component spins up a MoveNet multi-person detector inside a "use client" boundary, draws skeletons to <canvas>, and tracks per-player state machines (arms-up frames, wrist velocity, cooldown timers). We also used a Gemini mini-analysis path for increased accuracy..
Scoring logic: Each gesture produces a shot-type hypothesis (dunk, layup, normal). When the official feed reports the next possession, we compare timestamps and award points. Internally, we weight predictions via
$$ \text{confidence}{t} = \lambda \cdot \text{poseScore}{t} + (1-\lambda)\cdot \text{contextScore}_{t-1}, $$ which let us blend motion sharpness with previous drive context.

Challenges

Pose latency vs. UX polish: WebGL inference sometimes spiked past 25 ms/frame. We added a buffer plus cooldown tracking so duplicate gestures didn’t fire, and deferred heavy DOM updates with refs.
Data synchronization: NBA play-by-play IDs occasionally arrives far faster than what our streamed games were showing: we had to use the game clocks as well as action estimation and client side-customization to fix it.
Browser privacy hoops: Different laptops throw different permission prompts, so we built a “ready” banner and memoized video streams to avoid re-asking for the camera on every route transition.
TensorFlow bundle size: Keeping Tensorflow, wasm, and webgl backends in check required dynamic imports and a dedicated components/WebcamGestureDetector.tsx so the rest of the app stayed lean.

What We Learned

Treating pose estimation like any other real-time data source means building debuggable state machines, not just relying on raw model outputs.
Next.js App Router makes colocating UI + API logic pleasant, but live sports data still needs careful caching rules (we used cache: "no-store" extensively).
Human-friendly animations (multipliers, streak trails) make analytics digestible; we spent as much time on an engaging “feel” as on math.
Being explicit about fallbacks (TEST001 route, mock data) saves demos and keeps the gesture loop testable without live games.