ShelfSense | Devpost

ShelfSense Thumbnail
Receipt Scanning
Receipt Review
Item Overview
User Inventory
Recommendations
Notification

A HackUTD'25 Creation

🚀 Setup & Installation

Prerequisites

Python (v3.8+) and API keys for NVIDIA NIM, OCR.space, and API-Ninjas

Backend Setup (Run from `backend/` directory)

cd backend
python -m venv hackutdvenv
source hackutdvenv/bin/activate  # Windows: hackutdvenv\Scripts\activate
pip install requirements.txt
# Add your API keys to .env:
# NVIDIA_API_KEY=nvapi-xxxxx
# OCR_SPACE_API_KEY=K12345678
# NINJA_API_KEY=xxxxx
uvicorn main:app --reload --host 0.0.0.0 --port 8000

Frontend Setup

cd frontend/recipe-frontend
npm install
npx expo start

Testing: Visit http://localhost:8000/docs for interactive Swagger FastAPI documentation

💡 Inspiration

Food waste is a massive global problem - billions of pounds of food are thrown away each year just because people forget what they have or can't figure out how to use ingredients before they end up expiring. Our team wanted to create a solution that not only reduces food waste but also makes cooking more creative and accessible, leveraging the SoTA NVIDIA Nemotron supplemented by highly-interactive frameworks.

We created ShelfSense — an intelligent food management system that turns your expiring ingredients into delicious recipe opportunities, powered by NVIDIA's Nemotron llama 3.3 49b model as well as computer vision technology 🌱

⚙️ What it does

ShelfSense is a comprehensive food consciousness platform that helps users minimize waste through smart recipe recommendations.

Here's how it works:

Receipt Scanning - Users scan grocery receipts after a trip to the store using OCR.space API to automatically populate their virtual fridge and pantry
Smart Inventory Management - The system automatically tracks food items with quantities, expiration dates, and remaining shelf life using SQLite database
Expiration Detection - ShelfSense automatically identifies items with ≤25% of their shelf life remaining
AI Recipe Generation and Recommendation - Uses NVIDIA's Llama-3.3-Nemotron model to create personalized recipes featuring expiring ingredients, the model also queries APIs and databases itself to inform it's decisions
Fallback Recommendations - API-Ninjas provides additional recipe options when AI generation needs supplementation

All processing happens automatically — users simply scan receipts and get intelligent recipe suggestions when food is about to expire.

🛠️ How we built it

Backend: Built with FastAPI in Python, featuring three main modules: receipt scanning, inventory management, and AI-powered recipe recommendations
Database: SQLite with SQLAlchemy ORM managing food and fridge tables with expiration tracking
APIs & AI:
- OCR.space API for receipt text extraction and product identification
- NVIDIA NIM API with Llama-3.3-Nemotron for intelligent recipe generation
- API-Ninjas for supplementary recipe data and fallback options
Architecture:
- RESTful API design with comprehensive CRUD operations for fridge management
- Automatic expiration detection using mathematical formulas (duration ≤ expiration * 0.25)
- JSON-structured AI responses for consistent recipe formatting
- Environment-based configuration for secure API key management

This architecture enables real-time food tracking and intelligent recipe suggestions based on actual inventory data.

🚧 Challenges we ran into

OCR Accuracy: Fine-tuning receipt scanning to accurately extract food items from various store formats and receipt layouts
AI Prompt Engineering: Crafting prompts for Nemotron to generate diverse, practical recipes that actually use the expiring ingredients
Expiration Logic: Developing the right formula to determine when food is "about to expire" based on remaining shelf life percentages
Database Schema: Designing tables that efficiently track both original shelf life (expiration), remaining time (duration), as well as user inventory
API Integration: Orchestrating multiple external APIs through Nemotron while handling rate limits and fallback scenarios

🏆 Accomplishments that we're proud of

Built a complete food waste reduction system with automatic expiration detection in 24 hours
Successfully integrated multiple AI and OCR services into one cohesive workflow
Achieved zero-input recipe recommendations - the system automatically detects expiring food without user intervention
Created a robust database schema that accurately tracks food lifecycle and expiration status
Designed intelligent fallback systems ensuring recipe recommendations even when primary AI services are unavailable

Most importantly, we created a tool that could genuinely help families all around the world reduce food waste and save money while discovering new recipes.

📚 What we learned

The importance of mathematical precision in expiration calculations for food safety
How prompt engineering dramatically affects AI recipe quality and ingredient inclusion
Strategies for multi-API orchestration with proper error handling and fallbacks
The value of automatic detection systems over manual input for user adoption
How database design impacts the entire application's ability to track complex relationships

🚀 What's next for ShelfSense

Mobile App Development with React Native for seamless receipt scanning via camera
Nutritional Analysis showing health benefits of using expiring ingredients
Meal Planning Integration to optimize ingredient usage across multiple meals
Store Integration to automatically import purchase data without manual receipt scanning
Community Features allowing users to share creative recipes for common expiring ingredients
Waste Tracking Analytics to show users their environmental impact and savings

Our vision is to make ShelfSense the essential tool for conscious cooking — helping families waste less, save more, and eat better.

📁 Project Structure

ExpiredFoodSaver/
├── README.md
├── backend/
│   ├── main.py                    # FastAPI application entry point
│   ├── app/
│   │   ├── api/
│   │   │   ├── scan.py           # Receipt scanning with OCR.space
│   │   │   ├── updateddb.py      # Fridge inventory CRUD operations  
│   │   │   ├── recommend.py      # AI recipe recommendations
│   │   │   └── __init__.py
│   │   ├── database/
│   │   │   ├── db.py             # SQLAlchemy models and database setup
│   │   │   ├── food_tracker.db   # SQLite database file
│   │   │   └── __init__.py
│   │   └── __init__.py
│   ├── .env.template             # Environment variable template
│   └── .env                      # API keys (not in repo)
├── hackutdvenv/                  # Python virtual environment
├── frontend/                     # Frontend development (in progress)
└── .gitignore