QuickCall911

QuickCall911 is an AI emergency dispatch agent that uses voice recognition and NLP to handle emergencies, supporting 16 languages, and display real-time summaries for fast, efficient response.

Comment 1

Inspiration

Emergencies are incredibly stressful, and sometimes just a few seconds can make the difference between life and death. In multiple U.S. cities, nearly 30% of 911 calls are placed on hold for more than 20 seconds. In addition, many call centers are not equipped to handle non-English speakers which can waste even more time. I experienced this firsthand when a family member had a medical emergency but struggled to communicate because they didn’t speak English well. That’s what inspired us to build a tool that helps non-English speakers connect with emergency services and supports situations where there aren’t enough dispatchers to answer every call right away.

What it does

QuickCall911 is an AI emergency dispatcher that can be deployed when there is not enough human dispatchers. QuickCall follows a guide similar to human dispatchers and gives advice to people and then when a human dispatcher is available it will get switched out and will send a summarization of the conversation to the human dispatcher to get caught up. QuickCall is also proficient in 16 languages so that it can meet the needs of people who don't speak English.

How we built it

  1. Core Voice Agent Setup
    • Created a Python-based voice agent implementing OpenAI's Whisper API for transcription
    • Set up text-to-speech capabilities for responses
    • Created conversation flow with AI response trained for emergency dispatch
    • Added 16 language capabilities
  2. Retrieval Augmented Generation (RAG) system
    • Created a knowledge base with emergency protocols, location data, and medical guidelines
    • Set up semantic search capabilities for retrieving relevant information
    • Configured response generation with specific emergency handling protocols
  3. MongoDB integration for storing conversation history
    • Implemented call summary generation functionality
  4. Web Interface Development
    • Created an Express.js server for the web interface.
    • Built a website that can display MongoDB updates in real time.

Challenges we ran into

  1. We had trouble with Twilio because the free version did not allow for conversational responses on the phone, so we had to pivot and host the model locally.
  2. We also ran into trouble with the prompt engineering and getting the AI to give logical and timely responses
  3. Making the TTS agent work quickly for long caller responses

Accomplishments that we're proud of

  1. We built a RAG to improve upon the existing LLM.
  2. Getting multilingual support for the agent
  3. Allowing the entire project flow to work in real-time (making it useful in a real emergency)
  4. Implementing the seamless transitioning to human dispatchers
  5. The UI

What we learned

  1. How to implement RAG with a voice agent
  2. Creating voice agents that can speak multiple languages
  3. Working with Twilio API

What's next for QuickCall911

  • Improve RAG functionality
  • Make the voice agent sound more human-like
  • Branch out to more languages

Built With

Share this project:

Updates