AI assisted grocery shopping

Grocery Intel — AI agents collective that scans grocery store flyers, indexes deals in Elasticsearch, and finds you the cheapest basket across local grocers in seconds.

Comment

Grocery Budget Intelligence Agent

Problem: American families spend an average of $270/week on groceries — up 25% since 2020. Grocery pricing data is fragmented across chains, buried in ephemeral weekly flyers, and never systematically analyzed. Shoppers can't easily answer: is that "sale" actually cheaper than last month? Which store has the cheapest eggs near me? If I need five items for a recipe, should I go to one store or split across two?

Solution: Grocery Budget Intelligence is a multi-agent system that ingests weekly grocery flyers, extracts and indexes product pricing data in Elasticsearch, and answers natural-language shopping questions with real store recommendations — complete with fake sale detection and geo-aware store mapping.

How it works: Flyer images are processed through Google Gemini's vision API for OCR and product extraction, then normalized and indexed into Elasticsearch with geo-point mappings. Three Elastic Agent Builder agents collaborate to deliver grocery intelligence:

  • Deal Finder Agent searches current prices across all chains using custom ES|QL and index_search tools, ranks results, and recommends the cheapest store with address and zipcode.
  • Price Analyst Agent automatically verifies every recommendation against historical price data, catching "fake sales" where an advertised discount hasn't actually dropped below recent regular prices.
  • Flyer Extractor Agent validates and normalizes Gemini-extracted product data — standardizing names, fixing OCR errors, and categorizing products.

The agents are backed by 12 custom tools (4 index_search + 8 ES|QL) across 6 Elasticsearch indices. A Micronaut (Java 17) backend orchestrates the pipeline and serves a real-time dashboard with a Leaflet.js map, AI chat panel, and inline Chart.js visualizations via WebSocket.

Features we liked:

  1. Inter-agent cooperation — The Deal Finder automatically triggers the Price Analyst to verify recommendations. Both agent responses are combined with colored badges showing which agents contributed, making the multi-agent flow transparent. This plan-execute-verify pattern catches deceptive pricing that a single agent would miss.

  2. ES|QL tool accuracy — Embedding explicit index schemas (field names, types, index names) directly into agent instructions dramatically improved tool usage. The agent generates correct ES|QL queries on the first attempt rather than exploring the schema through trial and error.

Challenge: Extracting structured data from grocery flyers was harder than expected — prices hidden in semicircles, products bleeding across columns, wildly inconsistent layouts across chains. Unit price normalization was essential: one store prices chicken "per lb" while another prices a "3 lb package." Historical tracking also revealed chains cycling the same "sale" price every other week — a permanent price disguised as a deal.

Impact: For a family spending $270/week, even 10% savings from smarter store selection represents $1,400/year in real household savings.

Built With

  • 12-custom-tools)-elasticsearch-(6-indices-with-geo-point-mappings
  • chart.js
  • elastic-agent-builder
  • elasticsearch
  • google-gemini-vision-api
  • leaflet.js
  • micronaut
  • websocket
Share this project:

Updates