Project Hawkeye

AR tracked arena, displaying the goal and boundaries for the robot to interact within
Robot with his little hat (ArUco marker for self idenfication)
Robot in the middle of its course, marked by ArUco markers at every corner
Up close shot of the main character itself (shortly after being built)
Robot standing proud after completing successful navigation of location goal

Inspiration

Car's like those that are apart of Waymo's fleets, are as cool to look at as they are expensive. With all their components like lidar, radar, and all the camera's within them, their creation, maintenance, and operation is a financial drain. But what if it didn't need to be. What if there was a way to make autonomous vehicles cheaper to develop and maintain, while keeping (if not improving) their navigation. These are some of the things we aimed to accomplish for this project.

What it does

We've developed a end-to-end pipeline that takes real-time feed from camera/s (in our case, from iPhone's) from a developed and locally deployed app on our phones, to another local desktop app, that processes the information into navigational commands, and then sends them to our custom-built car. It is an autonomous process that gets us from point A to point B in the best way possible.

How we built it

We started with 2 local apps, the 1st built for iPhone's using swift, to take in real-time live feed and turn it into coordinates that can be interpreted on a 2D plain. The 2nd we built using X-code for pc's, that processes the coordinates from the first app, and then turns them into commands for the car to follow. Then we had to change where the inputs came from. We needed to have command inputs be given from a pc, instead of the remote control that belonged to the car. So, we flashed the firmware, added our own logic (including some commands for changing the colors on the RGB lights), and rerouted where the car was listening for input, to our pc's via the car's own Wi-Fi signal. Throughout this process we also used ArUco markers, 4 for each corner of the environment we want to navigate, 1 for the goal destination, and 1 for the orientation of the car itself. Thus we were able to create the end-to-end pipeline that allows for autonomous interpretation and navigation.

Challenges we ran into

From the beginning we knew this kind of project would be challenging. For 1, it was our entire team's first time working with hardware, so off the bat we're in unfamiliar territory. Then came the CV aspect of the project. What libraries would we use, where would the logic be processed (on the phone's or the pc's)? Then we had to deal with how exactly we'd process the resulting coordinates from the 1st app, into input that the car could use for navigation, via the 2nd app. Even after figuring out most of these challenges, we still had to test for accuracy, in both navigational pathing and obstacle detection.

What we learned

From this project we learned plenty. Personally I had concerns with handling hardware, but after constructing the car from scratch, and messing around with the firmware, I learned that, like everything in life, its not too bad once you've tried your hands at it once. As a team though, we learned how to use some of Apple's CV Libraries, general experience in programming in Swift and X-code, live-feed coordinate interpretation and conversion to navigational commands, and a combination of testing both the physical and software sides of a project.

What's next for Project Hawkeye

Though we've made incredible progress on this project in the limited amount of time we've had, we're not done yet. Some of the next immediate goals are further improving the navigational accuracy (including quicker paths and better obstacle navigation), expanding the scale of the testing environment, finding a better way of interpreting the 2D space we're working in (instead of fully relying on ArUco markers), and including support for multiple live-camera-feeds.