Baymax Bot

Inspiration

As students, we know how difficult it can be to look after yourself while juggling classes, deadlines, and living away from family. Two of our teammates are from Calgary, thousands of kilometres from home. Getting sick that far from home can make you miss the small things: someone checking in, reminding you to take your medication, or making sure you have what you need.

We watched Big Hero 6 as children and loved the relationship between Hiro, a teenage robotics genius, and Baymax, his loyal medical assistant. What stayed with us was how Baymax made care feel warm and personal. As robots enter our homes, we think those relationships will matter just as much as their capabilities. We wanted to explore that with BracketBot: could we build a friendly companion that helps with everyday care and makes being far from home feel a little less lonely?

What it does

Baymax Bot turns BracketBot into an interactive companion that can:

Respond to voice commands and hold conversations
Perform gestures such as waving, handshakes, fist bumps, hugs, and dancing
Run multi-step routines like a welcome sequence, calm moment, and dance party
Express itself using lights, sound effects, and music
Detect people and visible facial expressions using a fully local vision pipeline
Be controlled through an accessible web dashboard
Medical event checkers such as rPPG, fall detection, as well as a autonomous medical supplies packer.
Follow mode, so BaymaxBot never leaves your side.
Stop actions immediately through a global emergency stop control

Every physical action passes through a fixed allowlist and deterministic safety checks. The language model can suggest an action, but it never directly controls the robot's motors.

How we built it

We built BaymaxBot in Python on top of BracketBot OS. We created a local web dashboard that connects to the robot over SSH, bringing gestures, lights, sounds, music, and movement controls into one place.

To give Baymax personality, we used saved arm movements for gestures like waving, hugging, and fist bumps. We combined these with lights and audio to create welcome sequences, calm moments, and dance routines. Before playing a gesture, the robot checks that it is upright and limits how far its arms can move to reach the starting position. It then smoothly enters the gesture and returns to its original pose.

For voice interaction, we used Whisper to recognize speech locally, OpenRouter to generate conversational replies, and local text-to-speech to speak them aloud. Movement requests pass through a fixed list of supported commands and safety checks, keeping the conversational model separate from direct motor control. We also added spoken reminders that are saved locally and can run while the robot is busy with another task.

Our local vision system uses YOLO to detect people and body poses, YuNet to locate faces, and EmotiEffLib models to estimate visible facial expressions. Follow mode uses depth-camera data to track a person and maintain a chosen distance. It stops moving forward when an obstacle blocks its path and stops following when tracking becomes unreliable.

We used BracketBot OS’s existing SLAM and mapping system to track the robot’s position in its surroundings and support navigation. We connected this to our body-pose and depth detection so a possible person lying on the ground could be located on the map. We also added a rule that cancels autonomous navigation when this detector raises an alert. Alongside this, we built a camera-based pulse-estimation prototype that measures tiny changes in skin colour. These care-related features remain early prototypes that need further physical testing.

For medical-supply packing, we added an interface for a trained robot-control model. It sends camera images and current arm positions to a model server, then applies the returned arm movements to support pick-and-place tasks. The dashboard can coordinate the packing task while Baymax continues talking, blocking competing gestures from taking over the arms.

Finally, we built a simulation mode to test dashboard controls, routines, progress updates, and cancellation without connecting to the physical robot. A shared dashboard stop control requests cancellation of active tasks, while each controller handles its stopping procedure.

Challenges we ran into

Needing to Pivot: Our original idea was to have the robot perform origami, but we quickly discovered that its movements were not precise enough to reliably complete the folds. We had to pivot our idea and adapt the robot’s capabilities to something more achievable within the hackathon timeframe.
Connecting the Robot to the Internet: Network connectivity was a major challenge. With hundreds of hackers using the University of Waterloo network during Hack the North, the connection was often unstable. This made SSH access to the robot unreliable and slowed down development, testing, and deployment of changes.
Training the Robot: Collecting video training data was difficult because we initially had limited access to open spaces and varied environments. This made it harder to gather diverse training examples and ensure the robot could perform reliably under different conditions.

Accomplishments that we're proud of

We’re proud of giving BracketBot a personality. Combining conversation, gestures, lights, and sound made simple interactions, like a wave, a fist bump, or a friendly check-in, feel more personal. It brought us closer to the Baymax we imagined growing up.

Our biggest technical accomplishment was getting the different systems to work together. We connected voice interaction, local vision, depth-based following, and SLAM-based positioning, and built a dashboard to coordinate them. We also created a way for packing tasks to run while conversation and reminders remain available, without competing gestures taking over the arms.

We’re especially proud of building safety checks into those interactions. Spoken commands go through a supported action list, gestures have checks before movement, and the dashboard provides a shared stop control. Our simulation mode also let us test routines and cancellation without moving the physical robot. Getting these pieces to work together during a hackathon was a big achievement for our team.

What we learned

We learned that robotics is less about making one movement work and more about making every transition, interruption, and failure predictable. Getting one action to work amidst other actions that could be conflicting was challenging and was a great way to get introduced to robotics.

We also learned why conversational AI and motor control need clear boundaries. Letting the AI handle conversation while separate code checked and executed supported movements helped us give Baymax personality while keeping its actions controlled.

We also learnt through working with depth cameras and SLAM that understanding a room is harder than it looks. We had to collect data at different distances and check how the robot interpreted what it saw. Something that worked with a person standing directly in front of the camera could behave differently when they moved farther away or became partly hidden. Simulation helped us test the software, but testing on the robot showed us things we would have missed.

Finally, we learned that small details matter when building a companion. The timing of a reply, a gentle movement, or a well-placed sound could change how an interaction felt. We spent a lot of time getting features to work, but making them feel friendly was its own challenge.

What's next for BaymaxBot

If we ever take BaymaxBot further, we would need to take the same approach BracketBot took with their current line of bots and build this for a specific type of user. For instance, for students looking to have a virtual companion, we would want to add more natural speech interruption and turn-taking, consent-aware vision, authenticated remote access, and locally stored user preferences with clear privacy controls.

We also plan to expand Baymax Bot with safe navigation, obstacle detection, additional expressive gestures, and robot health telemetry. Eventually, we want it to combine conversation, perception, and approved actions into helpful routines while remaining transparent, interruptible, and safe around people.