Guardian | Devpost

Wearable AI to Trigger Calls/Calls and Send into to Dashboard
GIF
Landing Page
GIF
Map Dashboard
User Characters and Alert Data on the Map
Heat Map of Crime Hotspots
Text sent to Guardians
GIF
MVP Hardware Prototype

Problem Statement

In today’s fast-paced urban environment, family, friends, and emergency responders face significant challenges due to a lack of real-time visual and auditory information. Traditional emergency and SOS systems, such as Life360 and standard 911 calls, primarily rely on voice communication, text descriptions, or geolocation data. However, these methods often lack the situational context needed to fully understand the severity of an incident, placing limitations on what individuals in distress can effectively communicate. Check out our case study below for our stats and modelling.

Introduction

Guardian is a wearable AI that provides a comprehensive solution integrating real-time video and audio streaming with emotion and object detection analytics. If activated, this data is sent to a dashboard while simultaneously alerting emergency responders and emergency contacts with information about your surroundings and possible threats. By offering a unified platform for live updates and alerts, our project enhances and enables situational awareness—your personal guardian and SOS button.

Purpose

Guardian is designed with several key objectives in mind to address the limitations of traditional emergency and SOS systems:

Wearable: Guardian is a discreet and portable device that can be worn comfortably, ensuring that it is always within reach when needed. This increases the likelihood of timely activation during emergencies.
Hands-free: The system is designed to operate without requiring the user to use their hands, allowing them to focus on the situation at hand. This is particularly important in high-stress or dangerous scenarios where manual operation may not be feasible.
Capturing Live Data: Guardian continuously captures real-time video and audio data, providing up-to-the-minute information on the user's surroundings. This ensures that emergency responders and contacts receive the most current and relevant situational information.
Data-Driven Insights: The data collected by Guardian can be used to identify patterns and hotspots, contributing to broader public safety efforts and potentially preventing future incidents.
Seamless Integration with Emergency Services: Guardian's ability to contact emergency services directly ensures that help is on the way as soon as the device is activated. This integration streamlines the process of seeking assistance in critical moments.

How we built it

Hardware

With a Raspberry Pi, we capture video and audio footage using a webcam triggered by a keypad. The keypad serves two functions: enabling Guardian's webcam and contacting 911. Using scripts, the Raspberry Pi then converts the real-time footage into frames with OpenCV to meet machine learning model requirements.

Backend

We built a deep audio classification CNN model using TensorFlow to classify audio clips as either violent or non-violent based on their spectrograms. This model was trained on over 1000 entries. We also fine-tuned a Hume Expression Measurement model to isolate scores for "aggression," "hostility," and "frustration," outputting a JSON file with the results.

For visual context, we use Gemini flash to identify critical elements from the frames captured by the Raspberry Pi. Authentication and security are managed with Auth0 to ensure secure access to the system.

All collected data is stored in a MongoDB database, allowing for efficient and scalable data management.

Frontend

We used React, Mapbox, and Three.js to create visual representations of Guardian alerts and crime hotspots. The frontend provides a user-friendly interface for monitoring real-time alerts and viewing historical data on crime hotspots. Users can keep tabs on their loved ones with a 3D informational hub, offering up to date information on emergencies and whereabouts.

Challenges we ran into

Implementing the Hardware and Integrating the Raspberry Pi

Compatibility: Ensuring all components worked seamlessly with the Raspberry Pi.
Real-time Processing: Optimizing video and audio capture despite limited processing power.
Power Management: Balancing performance and power consumption for extended operation.

Integrating Backend, Frontend, and Hardware Components

Communication: Establishing reliable communication between all components.
Data Synchronization: Ensuring accurate, real-time data synchronization.
Security: Implementing secure authentication and encryption with Auth0.

Fine-tuning and Labeling Datasets for Violent Sounds on Hume AI

Dataset Labeling: Accurately labeling violent and non-violent audio clips.
Model Training: Achieving high accuracy in sound classification.
Emotion Detection: Fine-tuning the model to detect specific emotions.

Accomplishments that we're proud of

Hardware

Successfully triggering different functionalities through the keypad, including video capture and emergency calls.

Frontend

Creating an intuitive and interactive user interface for monitoring alerts and visualizing data.

Backend

Developing and integrating robust machine learning models for audio and visual classification, and ensuring secure data handling with Auth0 and MongoDB.

What we learned

We learned how to integrate multiple technologies, including hardware, machine learning models, and secure data management systems, to create a cohesive and effective solution. Here are some key implementations:

MongoDB

Scalable Data Management: We learned how to leverage MongoDB’s flexibility and scalability to manage large volumes of real-time data. This included setting up efficient data schemas for storing video frames, audio clips, and emotion analysis results.

Auth0

Secure Authentication: Implementing robust authentication mechanisms using Auth0 to protect user data and ensure that only authorized individuals can access sensitive information.
User Management: Utilizing Auth0’s user management capabilities to handle user roles and permissions, ensuring that different users (e.g., family members, emergency responders) have appropriate access levels.

What's next for Guardian

Using Guardian we can :

identify hot spots based off of real-time alert ping locations
understand emotional cues, assisting emergency responders and guardians in approaching situations
create a interactive mapping ability that further showcases "safety spots" such as fire stations, police stations, and hospitals
integration into purses

Case Study

Conducted a case study of crime hotspots, crime types, and crime per neighbourhoods int Toronto data to fully gauge the problem we are trying to target. Using simple plotting and linear regression models, we took data scraped from the web that identified key statistics:

30% of 911 calls could benefit from enhanced situational awareness
NCMEC reports that inclusion of video footage in Amber Alerts can improve recovery rates by 35%
The Bureau of Justice Statistics reduced average search time by 40%
Showing video footage through social media and public alerts increase the number of tips and leads by 50%

Modelling: https://colab.research.google.com/drive/1M5Ju4ssnh95Qx5bcAweBThjRuZJ34oK4#scrollTo=-_HqXzPZi1eC Scraping tool: https://colab.research.google.com/drive/1iqFYmSIcjy4WgLwTcaR20H9Fcx-SL0lM#scrollTo=Fq4EGEsXqJf1

Deployment:

https://guardian-safety.vercel.app/

Note: we have not deployed the backends yet! The backends are still local hosted. This is why a lot of the functions aren't working at the moment. Everything (all features) can be found local hosted.