Inspiration
Recently it's become a trend to put a fruit fly's brain in charge of things: playing video games, trading crypto, even suffering through virtual torture. We wanted to see what a fly's brain is actually capable of, and whether it could learn to play Devil Daggers, a fast first-person shooter. (We also had it play a scratch-ticket game, Scritchy Scratchy.)
What it does
Fly Daggers records you playing Devil Daggers as training data. A simulation of a real fruit fly's brain then learns from those recordings, by trying to predict what you pressed based on what was on screen. Once trained, the fly plays on its own: whenever Devil Daggers is open and a run is going, it watches the screen, and its brain presses the keys and moves the mouse. When it dies it restarts automatically and tracks how long each run lasted.
How we built it
The brain uses FlyWire's map of every connection in a fruit fly's brain, and a published model (Shiu et al., Nature 2024) that simulates how its neurons fire. Simulating 138,000 neurons is slow, so we wrote our own simulator that only computes the neurons active at any moment. That made it fast enough to play a live game on a normal laptop, with no graphics card needed.
We turn each frame of the game into the signals a fly's eye would send, such as bright objects, things getting bigger, and movement, and feed them into the fly's real visual neurons. An evolutionary algorithm tries many settings for the eyes and keeps the ones that let the fly's brain best predict what the human player did. A Windows program captures the game screen, runs the brain, and sends key presses and mouse movement back to the game. To check whether the real brain actually helps, we compare it against a blind fly, a fly with randomly rewired connections, and no brain at all.
Built with Python, NumPy, numba, and the Windows API.
Challenges we ran into
The newer, more complete fly brain from Janelia (MaleCNS) includes the ventral nerve cord, the fly's version of a spinal cord. It's much larger, and the brain model hasn't been validated on it yet, so we didn't have time to get it working. Instead we used FlyWire v783, which has the brain but not the nerve cord. Training was also slow: we tried to move it to Vultr cloud servers, but we couldn't get servers with enough CPU cores to train enough agents in a reasonable time, so all training ran on our laptops. We also didn't collect enough training data, since someone has to actively play the game for a long time; we'd want 2–3 hours and had about one.
Accomplishments that we're proud of
We were successfully able to train the fly brain to play Devil Daggers and train it without using any GPU processing and were able to optimize the code to run much faster. We also launched a website that has a live stream of the fly playing the game and other information about the project.
What we learned
Teaching the fly is hard and very time consuming, it was accurately repeating the players movements 5% of the time. When we first gave the fly control of the game, it kept killing itself, most likely because it realized the player was holding "w" most of the time.
What's next for Fly Dagger
Record far more training data for the fly to learn from. Instead of teaching the fly to learn from player movements, we should use reinforcement learning to teach it how to properly play the game and reward it for completing certain actions like staying alive or being accurate. We plan to use the far more complicated brain model from Janelia MaleCNS so that the flys outputs drives its motor neurons.
Log in or sign up for Devpost to join the conversation.