rainbolt.ai

Bolt across the world with rainbolt.ai!

Comment

Summary

Ever see a pic and wonder: “Damn, this place looks cool. Where is that?”

Sometimes, it's difficult to get any accurate results at all. So, we made rainbolt.ai. Here to help you discover your next dream vacation spot, find the location of your long lost cousin's photo, and to simply help you learn about geography and culture around the world.

(Name inspired by Trevor Rainbolt)

What it does

  • Rainbolt.ai allows users to upload any image of a location, and identifies the location with confidence. For each match, nearby street view images are displayed from Mapillary, a link to the coordinates on Google Maps is generated, and facts and info are displayed about the culture and features of the location.

  • Rainbolt shows its reasoning process— recognizing road markings, languages and scripts, signage, architecture, vegetation, plate formats, and map-matching checks— allowing users learn about new geographic regions and features specific to them.

  • Users can interact with the agent for clarification or to receive a new set of matches if deemed to be inaccurate.

  • Organize your learning and thoughts with our constellation feature. Visualize chats on a dynamic whiteboard and link those with related subjects.

How we built it

RAG Pipeline

Pillow - Extracts images from dataset documents OpenAI CLIP - Processes and encodes image and feature dataset with over 900k objects (sourced from the YFCC100M and geoguessr guides)

Pinecone - Vector database for efficient embedding storage and querying (top-k)

LangChain - Orchestrates the AI pipeline and manages LLM context/prompting using queried results from database

Gemini API - Powers reasoning, coordinate approximation, and content generation

Mapillary API - Searches the Mapillary database for street view photos in similar coordinate locations

Backend

FastAPI - High-performance Python backend for handling API requests and creating websockets

Frontend

Next.js - React framework for server-side rendering and routing

Tailwind CSS - Easy and sleek styling for responsive UI components

Zustand - Lightweight state management for React

Data/Authentication

Firebase - Real-time database storage for user and session IDs

Auth0 - Easy and secure authentication and user management

Deployment

Google Cloud - Scalable cloud infrastructure for hosting and deployment

Challenges we ran into

Integrating three.js data visualization seamlessly without snappy animation, figuring out rotational axis number amidst camera movement. Deploying the website to Google Cloud. Dynamic Street-Level Vision Integration via Mapillary API. Optimizing Gemini output.

What we learned

Integrating auth0 with firebase backend. Good Github practices and commands. NEVER USE DEBIAN!

What's next for rainbolt.ai

Video Analysis Pipeline - Extend CLIP capabilities to process video feeds frame-by-frame for temporal understanding Real-time Object Tracking - Implement continuous monitoring of street-level changes using Mapillary's sequential imagery 3D Spatial Mapping - Generate depth maps and 3D reconstructions from multi-angle street imagery

Built With

Share this project:

Updates