Pi-Pal

Care so simple, it's just a wave away.
GitHub »

Cindy Li · Cindy Yang · Elise Zhu

About The Project

alt text

Imagine a hospital where patients can adjust their room's lighting, temperature, or even call for assistance—all with a simple gesture. Picture nurses administering medications to patients without physical contact, ensuring hygiene, speed, and efficiency. Welcome to Pi-Pal, a cutting-edge healthcare solution that blends gesture recognition technology with smart automation to redefine patient comfort and nurse productivity.

Problem

Hospitals often rely on manual controls and physical interaction for basic tasks, making them cumbersome for patients with mobility challenges or contagious illnesses. Traditional medicine dispensation requires physical contact, increasing the risk of contamination and exposure for both nurses and patients, especially during pandemics.

Our Solution

Pi-Pal introduces a gesture-controlled hospital room system powered by an advanced computer vision model and demo-ed with socket connections to a Raspberry Pi. Here's what it offers:

Contactless Environment Control: Patients can adjust lighting and call for assistance using hand gestures, tailored to their mobility.
Contactless Medicine Dispensal: Using gesture controls and facial recognition, nurses can unlock and dispense pre-assigned medications through a hygienic, automated system.
Efficient Workflow: Track all requests and actions in real-time with a streamlined UI, ensuring timely responses and efficient patient care.

Built With

Technologies

Computer Vision

Our solution leverages OpenCV and MediaPipe for real-time gesture and face detection, ensuring high accuracy and efficiency in dynamic hospital environments.

Gesture Detection: Using MediaPipe’s hand tracking module, we accurately identify and interpret patient gestures, such as holding up fingers to adjust lighting or raising a call sign to call for assistance. This ensures a seamless, intuitive interface that works even in low-light or cluttered environments.

Face Detection for Nurse Authentication: For features that required nurse approval, we wanted to adopt a certification system. So, we used OpenCV to provide a secure and contactless method for nurse verification for pill dispensing. We used Haar Cascade Classifier and LBPH (Local Binary Pattern Histogram) for real-time face detection, and trained our model on our faces to test for accuracy.

Efficiency with OpenCV: OpenCV handles video feed processing, enabling smooth real-time performance with minimal latency. Its integration ensures that gesture recognition operates seamlessly on affordable hardware, making the system scalable for widespread hospital deployment.

alt text

Hardware Pipeline

To bridge the gap between gesture recognition and physical hardware control, we utilized sockets to send real-time commands from our gesture detection system to a Raspberry Pi, which acts as the central hub for managing room devices.

Sockets for Real-Time Communication:

Our system uses Python sockets to establish a seamless communication channel between the gesture detection module (running on a laptop or server) and the Raspberry Pi. When a gesture is detected , the corresponding command is sent to the Raspberry Pi over a TCP connection. This setup ensures immediate responsiveness, making the interaction feel smooth and intuitive.

The use of sockets allows the gesture detection system to run on a powerful external machine, while the Raspberry Pi focuses solely on device control, reducing load and making the system highly modular.

This architecture makes it easy to add more devices or expand functionality without disrupting the core setup.

Hardware Integration with GPIO:

The Raspberry Pi is wired to control essential room devices via its GPIO (General-Purpose Input/Output) pins:

LEDs: Represent the lighting in the room; dimness can be finely personalized by adjusting the PWM (Pulse Width Modulation) signal.
Buzzer: Alerts nurses when a patient calls for assistance
Servo Motor: Enables contactless medicine dispensing by controlling a small 3D-printed mechanism to deliver medication trays directly to patients.

alt text

Analytics

Our system integrates with Databricks and OpenAI for real-time data processing and analytics, enabling hospitals to monitor patient requests, nurse responses, and room conditions in a centralized dashboard. All data from the backend servers are stored in MongoDB for easy access and retrieval.

We display key metrics on a user-friendly interface coded using Next.js and Tailwind CSS, ensuring that hospital staff can quickly access and act on critical information. This frontend is hosted on GoDaddy.

alt text