Inspiration
Waymo... Grandma.... Waymo... what if your grandma could also be a Waymo. Waymo's have lidar scanning and autonomous driving how cool would that be if something like a wheelchair that even your grandma could use had that technology.
What it does
Okay imagine this. You're someone who needs a wheel chair on a daily basis, you have to get around your house, do things like turn on the lights in the dark. All with accessibility issues. Now whats a creative way we can make this easier. We have Granny Waymo (Kevin) a robot that can perform tasks for you, like turning on the lights in the dark (with our lidar scanner) and using its automatic movement to move and press the light switch itself. We also have an app that allows you to speak to it in live time making it possible to ask for Granny Waymo (Kevin) to grab you a cup of water without having to even raise an arm.
How we built it
Codex & OpenAI API
Codex:
- We pushed hella features by using Codex IDE. We spun up many threads, told the agents to watch out for other agents and to collaborate together. The agents then started chatting and updating other agents in the threads, keeping our changes and features commit free. This allowed us to build at lightning speed.
- We also enabled remote control, which allowed us to talk to sponsors, wait in long food lines, and attend workshops, all while coding and monitoring our agent's progress from the app.
- One pretty niche thing we used Codex for was this. Everyone in Robotics knows how time consuming calibrating parts were. However, we used Codex intelligently for this. We placed our Mac book on top of our robot, told Codex to calibrate the motors and stuff via the gyroscope, taking a picture from the webcam, accelerator, among other stuff. Within minutes it was able to fully calibrate itself with no extra prompting.
OpenAI API:
- GPT Live 1 - Used for personal communication with the robot, and having it delegate tasks via tool calls to allow us to converse with the robot live, while having it perform tasks for you. For harder tasks, instructions are routed to Astra.
- GPT Astra 6 - Main model for controlling the robot, leveraging it's spacial awareness and intelligent step planning.
Agent Harness (Rox)
Our agent works with the messiest data possible, raw sensor data from a live iPhone, lidar, geographical positioning, orientation, among others. We designed tools to help the agent cut through noise and easily stack and plan their next steps. These are the kinds of situations where agents NEVER have enough data to actually perform at their best.
- Cleaning the data before the model sees it. Raw lidar is useless to a language model, so it never sees any. We fuse it into an occupancy grid, inflate every obstacle by the robot's radius, and A* through it. The agent says "3 metres at 40 degrees" and gets back a clean answer: clear distances left, ahead, right.
- Validating every source. Frames older than 30 seconds get thrown out, pose older than 2.5 seconds is stale, and target sightings under 0.6 confidence are discarded. Bad data is dropped, not averaged in.
- Resolving conflicting sources. Three views of the same room disagree constantly: the live frame, the fused room mesh, and what the cameras recorded earlier. We label them for the agent, the frame is truth, the mesh is memory, the recall is "hints, never proof" so it knows which one to believe and when to go look again.
- Deciding under uncertainty. iPhone pose lags by up to a second, so our first 60 degree turn overshot to 153. The robot stopped trusting the sensor and learned it instead, re-planning every 2 metres and firing a corrective turn when it's off by more than 15 degrees.
- Errors as instructions. A failed tool doesn't rand returns "Wheels stopped. Call look and continue." Every tool hands back a fresh frame and map, so the agent recovers by looking instead of guessing.
The Lore
We uh, we leveraged facebook marketplace LOL, jk, kinda not really, we were missing a motor for our wheel chair, and we had to find one that matched our battery size and power requirements. So we had to scour facebook marketplace for anything that potentially had something of the size thats when we found a, toy Kawasaki 4 wheeler and we decided to drive over, pick it up, and then saw it apart... literally, we used the power tools provided in the maker space and we sawed it apart (check the attached photos) and thats how we got our two motors and the two massive tires on our robot. The rest was a little less of an adventure but still a challenge, we had to 3D print a ton of custom parts to fit everything together including our robotic arm, took over 20 hours of 3D printing (not including the failures) than we had to build an app, a dashboard, and hack into the Hack the North badge to use it as a controller.
That was its own whole adventure honestly, because the badge kept bricking on us, its meant to show you your schedule not really to drive a robot. So we pulled the stock firmware off it over USB-JTAG and sat there reading through the disassembly to find the pin map, which is how we found out the buttons arent even wired to the chip directly they all come through a little shift register, and the screen setup was buried in there too. Then we wrote our own firmware for the thing and now it's got a D-pad that actually drives, a screen showing you battery and signal and what mode you're in, and the LEDs change colours according to how it should work, so green when its movin, and red when it stops.
On our software side, we also built our agent by splitting it into two brains basically, one that talks and one that actually does stuff. The talking one is the new gpt live and it only handles the conversation, it listens, it goes mhm and yeah while you're still talking, and if you cut it off halfway it shuts up and listens which sounds like the easiest thing in the world but took us way longer than we want to admit. Then sitting behind it is an agentic system, which is built off of codex... like we literally forked codex, and then completely redesigned it to make it work with our bot, it can now see (look around do some scans and stuff), search for an object, drive the base, set the arm, run the winches, and save things to memory so it remembers what you like next time you talk to it.
Challenges we ran into
This whole weekend was a ton of ups and downs, we literally didn't have the parts, didn't have the experience to build it, but we made it happen. From going on a facebook marketplace run to rebuilding the whole robot 3 times. Trying to hack into the hacker badge it was all challenges that we somehow managed to pull through. Also the wifi in this building was not wifiing but its all good, combined we burned over 100 gb in data this weekend too.
Accomplishments that we're proud of
Honestly? That it drives at all. We started with a wheelchair and no motor and we ended with a thing that rolls across a room when you push a d-pad on a conference badge, and that still feels a little insane to me. We're proud that it's a real robot and not a demo video, it's got real motors out of a real toy that we really did destroy with a saw, and real tires, and it will genuinely run over your foot if you're not paying attention. We're proud of the badge, taking something that's meant to sit around your neck and blink at you and turning it into an actual controller with a real E-STOP on it felt so good. And we're proud that the agent can't lie to us, which sounds like a small thing but so many agent demos are just the model going trust me bro I did it, and ours has to come back with a receipt before it's allowed to open its mouth. Also nobody got hurt, which given the amount of power tools and lithium batteries involved is genuinely on the list.
What we learned
So much of this was electrical and we learned that the hard way, like the actually painful way. We cooked a microcontroller, twice, and both times it was the same dumb thing, a servo power wire was sitting on the ground rail so 6 volts was running straight through the chip's ground pins and the chip was basically part of the circuit, it got hot enough to smell burnt. But the real lesson wasn't check your wiring, it was that we kept getting the exact same error over and over and kept poking at it like it was a software problem when the hardware had been screaming at us the entire time. If you get the identical error four times in a row, stop typing and go look at the board. We also learned to measure things instead of eyeballing them, we spent a while trying to get joint angles off a camera photo and it turns out you just can't, you need a real measurement or a marker or something you can actually trust, a photo will happily lie to you with a lot of confidence. And we learned to keep calibration in exactly one place, all the wheel trim and the sign flips live on the laptop so the badge and the phone and the dashboard aren't each carrying their own slightly wrong copy of reality.
What's next for Granny Waymo
The arm. The arm is the big one. We got it designed and printed and wired and then the board driving it died on us (went kaboom), so we had to rebuild it in a rush, and not have it super calibrated. We wish we could have the hand complete complex tasks for us, but that would be an addition in the future.
Log in or sign up for Devpost to join the conversation.