Plotholes

Plotholes uses AI and crowd-sourced data to detect, classify, and analyze road damage in real time—turning vision data into a live map that helps cities fix infrastructure before it fails.

Comment 2

Inspiration

Driving through college towns and cities, we noticed the sheer amount of road damage—from potholes to cracked asphalt—that impacts not only the comfort of driving but also safety and vehicle maintenance. We wanted to create a solution that helps with quicker repairs to road damage and more immediate driving comfort. By combining updated NYC 311 data with crowd-sourced, user-reported observations, our web app maps street conditions in real-time, visualizes and plots them on both interactive maps and heatmaps, and provides AI-assisted analysis to analyze repair data and road damage severity. Additionally, it helps drivers plan safer, smoother routes that avoid damaged streets.

Our goal is to transform the everyday frustration of dodging potholes into actionable insights for city planning, road maintenance, and improved driving experiences. What started as a personal annoyance turned into a drive to make roads safer and more efficient for everyone.

What it does

Plotholes is a lightweight geospatial analytics platform that extracts, aggregates, and visualizes open street-issue reports (potholes, hazardous road conditions, etc.) as heat maps and neighborhood-level risk layers. It then uses these insights to help users path plan a route between two points that avoids streets with recorded damage. Users can also self-report street damage with an image of the issue and the location on the web-app, and the CV model will analyze the image, assign a level of severity to the damage, and add report to the list. City planners and municipalities can use this data, along with Snowflake Cortex's generative AI tool to make more informed decisions about where to put their road repair efforts.

How we built it

Plotholes combines a Flask backend for data processing with a Next.js + Leaflet frontend for interactive map visualizations.

Backend Features

  • Data Ingestion & Storage: Pulls in NYC 311 data, crowd-sourced user reports, and image-based pothole detections, storing everything in Supabase and Snowflake.
  • Severity Analysis: Calculates the risk level of each street issue to prioritize repairs and highlight high-impact areas.
  • Geospatial Processing: Uses Shapely for point-in-polygon operations and neighborhood-level aggregation, producing GeoJSON for mapping and heatmaps.
  • Routing & Path Planning: Leverages OpenStreetMap to compute shortest paths and alternative routes that avoid damaged streets.
  • API Endpoints: Exposes endpoints for raw issue data, neighborhood summaries, and map-ready data for the frontend.
  • AI Integration: Incorporates a YOLOv8 pothole segmentation model (https://huggingface.co/keremberke/yolov8n-pothole-segmentation) from Hugging Face to automatically detect and classify street damage from uploaded images.
  • Natural Language Query Support: Uses Snowflake Cortex to convert user text queries into SQL, enabling flexible exploration of street conditions.

Frontend Features

  • Interactive Map: Displays potholes and street damage as point heatmaps and neighborhood-level risk overlays.
  • Layer Controls: Users can toggle between raw reports, heatmaps, and risk choropleths.
  • Path Planning: Computes alternative routes that avoid high-risk streets.
  • User Reporting: Allows users to submit new issues or upload datasets, updating the map in real-time.
  • Modular UI Components: Components like NYCMap, ReportDrawer, and UploadBox keep the interface clean, responsive, and extensible.

This stack lets Plotholes combine official data, crowd-sourced reports, AI detection, geospatial analysis, and interactive frontend tools to give users a real-time, actionable view of street conditions.

Challenges we ran into

  • Messy and Limited Data: We initially aimed to focus on areas like Amherst or Boston, but there simply wasn’t enough data to build meaningful insights. Switching to NYC gave us a rich dataset, but introduced its own challenges with scale and complexity.
  • Scaling and Performance: Handling NYC’s large dataset and area caused computations to be slow at first. We had to rethink our approach, implementing caching strategies and generating data asynchronously to keep the app responsive and low-latency for users.
  • Learning New Tools: Integrating new technologies pushed us out of our comfort zone. It was our first time working with a Hugging Face model for pothole detection, integrating Snowflake Cortex for natural language queries, and working with any kind of maps data, requiring quick learning and experimentation.

Despite these hurdles, overcoming them strengthened both our technical skills and our approach to designing scalable, real-world solutions.

Accomplishments We’re Proud Of

  • Built a full-stack geospatial platform that integrates NYC 311 data, crowd-sourced reports, and AI-analyzed images to provide a real-time map of street conditions.
  • Designed and implemented a severity scoring and neighborhood-level risk aggregation system, helping users and city planners prioritize repair needs.
  • Developed a modular, interactive Next.js frontend featuring point heatmaps, choropleth overlays, layer controls, path planning, and dataset upload/reporting functionality.
  • Integrated AI-based pothole detection using a Hugging Face YOLOv8 model, automatically analyzing images to enhance street condition reporting.
  • Leveraged Supabase, Snowflake, and Snowflake Cortex to store, query, and explore large-scale datasets, including natural language query support for flexible analysis.
  • Created a scalable, extensible framework capable of handling large datasets and adaptable for broader civic tech applications.

These accomplishments demonstrate our ability to combine data engineering, geospatial analysis, AI, and interactive frontend design into a cohesive tool that improves city infrastructure awareness and helps drivers avoid hazards.

What we learned

  • New tools! Hugging face, Open Street Maps, Snowflake Cortex, Supabase, and so much more :)

What's Next for Plotholes

  • Expand coverage to more cities and datasets.
  • Implement kernel density estimation (KDE) for smoother, continuous heatmaps.
  • Move geospatial processing to PostGIS for faster, indexed spatial queries.
  • Enable dynamic or ML-based severity scoring for smarter, predictive analysis.
  • Boost frontend performance with clustering, progressive loading, and large dataset support.
  • Enhance AI capabilities for more accurate and automated pothole detection and insight analysis.

Built With

Share this project:

Updates