flih | Devpost

Inspiration

Recently, a trend emerged where people would take the synapse-level brain scan of a fly, and train it to perform tasks of varying seriousness... We always wanted to develop an autonomous robot, and we thought to ourselves- what better to use for its brain, than a real brain?

Fly brain simulation

Fly playing a game

What it does

FLIH (Friendly Local Invertebrate Helper) is a wheeled robot that can navigate around environments. Onboard cameras and LiDAR enable a companion fly brain scan as well as Huawei's OMNI multimodal models for help. The fly brain aids with reactive avoidance while the OMNI models provide scene understanding. The project runs on a Jetson and RPI Pico 2, supported by an RC battery and a portable charger.

How we built it

HARDWARE We first built the frame out of available plywood and bolts, making two levels to separate sensors from electronics. The cameras were positioned at the front with strategic angles to maximize coverage, both horizontal and vertical, while the LiDAR sits behind at the highest position for clear visibility. A RC style battery is stepped down to power the motors, while a portable bank powers the Jetson and LiDAR.

BACKEND The fly brain connectome scan was downloaded from the Flywire dataset, and represents a scan of every connection between neurons. This model was trained in a MuJoCo sim environment on obstacle avoidance. Specific brain activation patterns are mapped to suggestions: i.e. move forwards, move forwards and left. Huawei OMNI model API was sourced from the challenge page, and are called on both the camera images and LiDAR. The model then returns a scene analysis and points out any obstacles in front.

FRONTEND The original vision was to have a fully functioning tool that allows users to call the robot to their location, and then be guided to where they needed to be. While our vision shifted, our front end still offered users the ability to control the robot through a bidirectional web-socket, being able to see directly from the eyes of the robot. Additionally, the front-end allows for in-depth training statistics of the fly-brain during its simulation, allowing users to track progress more effectively.

Challenges we ran into

The biggest source of our challenges was the fly brain. We quickly learned about how the brain is structured, how messages are passed, how information propagates, and how these processes translate into computers. But from there, the quirks of the resulting graph (a very sparse matrix with lots of loops) fought us, and throwing CUDA at the problem did not help. We were able to bring down time for a single optimization step from 8 minutes to around 20 seconds by reducing the transfers needed between CPU and GPU.

Fly training

Yet despite all this, training the model presented yet more challenges. Despite our best efforts, we did not have enough training time in our MuJoCo sim, and several sim-to-real gaps further degraded model performance. The last issue was deployment on edge hardware, where the lack of full-speed inference rendered the sensitive graph connections unusable. These issues compounded and resulted in a pivot to the fly brain as a supporting piece to a more traditional navigation system.

There were also numerous challenges during the hardware design, most stemming from the limited materials available. While cutting the plywood was easy, drilling accurate, consistent holes with screwdrivers was certainly not. The accurate placement and alignment of the sensors to ensure accurate sensor extrinsics was also a challenge, but managed with 15cm popsicle sticks. And making use of our RC battery’s odd adapter required a last minute trip to Home Depot for electronics.

Accomplishments that we're proud of

Despite all of the challenges, we were able to integrate a fly brain connectome and multimodal scene understanding on a Jetson, and we built an robust wheeled platform that gave us the freedom to experiment and pivot. The functionality of a web-based tele-op mode also made for fun demos.

Fly playing a game

What we learned

We learned many things during the development of FLIH. For one, since none of us had built a project like this before, we better understood how difficult it truly was to develop a true RL system. Though we had a good training pipeline and a full simulation built, the results were still fairly inconsistent and not good enough to fully navigate the robot on its own without user input, which was certainly a challenge. We also better learned how to develop on the Jetson hardware, as only one of us previously had experience developing on this platform. Finally, we also better learned how to innovate on the fly. We had a lot of challenges during development, and us being able to pivot consistently and still produce an end product was something truly new!

What's next for flih

Moving forward, we would love to expand FLIH to be fully driven by the fly connectome. With more time, our team would be able to construct a more robust RL training environment, both simulated and non-simulated, allowing the model to generalize better to the real-world environment. We also hope to have better map connectivity, as well as outdoor connectivity and GPS, to allow the robot to guide users both indoors and outdoors. Thank you for your continued support with FLIH!

flih out.