Reverse Recipe

Inspiration

We've all been there: you're out to eat, something lands on the table that's genuinely delicious, and you think "I want to make this at home" — but you have no idea what's in it or how it comes together. Recipe apps make you start from a name and search. We wanted to flip that around. Start from the photo — the thing you actually have — and work backwards to the ingredients, the macros, and the steps. Point-and-shoot food, reverse-engineered.

What it does

Take a photo of any meal and Reverse Recipe hands you back:

• the most likely name of the dish, with an honest confidence rating
• a full ingredient list with rough household amounts
• a nutrition panel — calories, protein, carbs, and fat for the whole plate
• servings plus prep and cook time
• clear, numbered cooking steps a home cook can actually follow

From there you can print the result as a clean recipe card, or drop into a hands-free Cook Mode: the app reads each step aloud in a natural neural voice and listens for spoken commands ("next step," "go back," "repeat") so you never have to touch a screen with messy hands.

How we built it

• Next.js (App Router) + React 19, deployed on Vercel. No database, no accounts, no tracking — the photo goes straight to the model and the result comes straight back.
• Google Gemini (gemini-flash-latest) does the vision and reasoning. A single API route sends the image with a strict JSON response schema, so the model reliably returns structured data we can render without babysitting.
• In-browser image resize — photos are scaled down on a canvas before upload so requests stay small and fast, which matters on a phone.
• On-device neural read-aloud — Cook Mode uses Kokoro, an 82M-parameter TTS model that runs 100% in the browser via WebAssembly. No paid TTS API, nothing leaves the device, and it sounds genuinely human. Audio is synthesized and played in chunks so a long recipe starts talking almost immediately.
• Web Speech API for voice command recognition, with whole-word phrase matching so "background" never gets read as "back."

Challenges we ran into

• Gemini kept returning empty responses. Flash "thinks" by default, and that reasoning eats the same token budget as the answer — paired with a JSON schema it would spend the whole budget thinking and hand back an empty candidate. Setting thinkingBudget: 0 was the single biggest reliability win.
• Pinned model versions kept breaking. Hard-coding a specific model (gemini-2.5-flash) meant the app 404'd the moment Google retired it for new keys. Switching to the -latest alias keeps us on a supported model with zero code changes.
• Getting a good voice for free. We tried server-side neural TTS (Edge, Gemini) — the free endpoints block datacenter IPs, so it worked locally and died on Vercel. Moving TTS fully on-device with Kokoro sidestepped the whole problem, and getting it to sound right meant running WASM + q8 weights instead of WebGPU, which produced garbled audio.
• Staying genuinely free without a hard wall when the shared quota runs out.

Accomplishments that we're proud of

• A real, useful result from a single photo — a legitimate starting point for a shopping list or a rough calorie count.
• A read-aloud voice that sounds human and costs nothing, running entirely in the browser with no API and no data leaving the device.
• Truly hands-free cooking: you can walk through an entire recipe by voice alone.
• Graceful free-tier handling — when the shared key hits its daily quota, the app doesn't just error; it invites you to paste your own free Gemini key, which is stored only in your browser and sent only alongside your own requests.
• A polished, appetizing UI with light and dark themes and a clean printable card.

What we learned

• Structured output (response schemas) plus turning off "thinking" is what turns a flaky vision call into a dependable feature.
• Prefer model aliases over pinned versions unless you have a reason not to — it saves you from silent retirements.
• The browser is a surprisingly capable ML runtime. A small quantized model over WebAssembly gives you a real neural voice for free, with perfect privacy.
• Free-tier UX is a design problem, not just an error case — meeting quota limits with a graceful path keeps the app usable for everyone.

What's next for Reverse Recipe

• Save & collect recipes you've generated, and share a card by link.
• Dietary adaptations — regenerate a dish as vegan, gluten-free, or lower-cal.
• Multi-photo & video input for more accurate portion and ingredient reads.
• Smarter portions — scale ingredients and macros to a target serving count.
• Export to shopping list and grocery/instacart integrations.
• More languages in both the recipe output and the read-aloud voice.

Built With

• gemini
• nextjs
• vercel