TowerGuard

Fatigue detection for Air Traffic Controllers

Comment 1

Inspiration

I was an intern at Delta Air Lines last year. My work involved analyzing employee injuries, and I found that many came from stress and fatigue. During the recent government shutdown, Air Traffic Controllers were working for weeks without pay in a job that requires unreasonable amounts of attention. I wanted to build a tool that would simultaneously help ATCs reduce mental load, help keep airline passengers safe, and airports be more efficient.

What it does

TowerGuard analyzes an Air Traffic Controller's fatigue levels using CV and a custom detection model I fine tuned using DigitalOcean. A neural network also analyzes a recording of the ATC's voice and its spectrogram to detect vocal fatigue. Then, Gemini provides a recommendation, spoken aloud by ElevenLabs. Snowflake's Snowsight dashboard tracks and reveals trends in the data, and Solana tokenizes each individual analysis. Everything is hosted on the cloud using a DigitalOcean droplet.

How I built it

A native WebRTC library is used to stream webcam data to the backend. A pretrained Haar Cascade Classifier detects faces and adds a bounding box. I used an NVIDIA H100 through a DigitalOcean GPU Droplet to finetune Google’s Vision Transformer (ViT) model pre-trained on ImageNet-21k using a Driver Drowsiness Dataset (DDD) dataset from Kaggle. My model then returns whether it thinks the face in the OpenCV environment is fatigued or not.

Find my model here: https://huggingface.co/jeremoo/vit-fef-finetuned

The WebRTC audio stream is first recorded to an MP3 file. Once the recording is stopped, the file is transcribed using OpenAI’s whisper-tiny model. Once the ATC starts and stops recording the file is transformed into a Mel spectrogram using the Short Time Fourier Transform (STFT). I fine tuned a HuBERT-base neural network to calculate and weight fatigue/alertness within the speech sample. This analysis, along with the fatigue status and transcription, is fed to the Gemini API to generate a final 'AI supervisor report’ with recommendations for the ATC. I used the ElevenLabs API to play the report back in real-time.

I moved TowerGuard into the cloud using a DigitalOcean CPU-Optimized Droplet running Ubuntu. This hosts our aiohttp Python backend. I configured Nginx as a reverse proxy to process incoming traffic and used Certbot to create an SSL certificate to use HTTPS on the domain I got using MLH’s code (iloveplanes.yoga). The backend runs as a systemd service and saves speech recordings and spectrogram images into DigitalOcean Spaces (S3-compatible) using boto3. Every transcription, fatigue status, weighting, and Gemini/ElevenLabs report is written to a Snowflake database. I wrote custom SQL scripts to track operator fatigue over time, through different parts of the day, and fatigue/alert percentages on a Snowsight dashboard.

Finally, I integrated the Solana blockchain as forms of ‘proof’ receipts. After the entire backend process completes, the server calls the Helius v1 Mint API which mints a new compressed NFT (cNFT) to my Phantom wallet. The NFT’s metadata is created using the fatigue/alert results, the Gemini report is the NFT’s description, fatigue status are set as on-chain attributes, and the spectrogram image and audio file are included as urls. Basically, this uses blockchain to verify the proof of each analysis.

Challenges I ran into

  • DigitalOcean’s Atlanta servers were down/not available this weekend which delayed model training
  • Was pretty difficult learning how Solana mints NFTs and how to integrate the API for each analysis
  • Lack of sleep…

Accomplishments that I'm proud of

  • Fine tuning my own model and publishing on HuggingFace
  • Submitting to every single MLH track
  • First solo hackathon :P

What I learned

  • Model fine tuning
  • Blockchain cNFT minting
  • Complex data pipelines with Snowflake

What's next for TowerGuard

  1. Hardware integration; I have decent embedded experience so it would have been fun to build a siren that blares when fatigue is detected
  2. Auth0 + PostgreSQL for logins and multiple users
  3. Platform to trade the cNFTs haha

Built With

Share this project:

Updates