Self Driving Car Simulator with Assetto Corsa

Inspiration

We were inspired to create this after seeing the Roborace driverless auto racing series. A lot of research has gone into driving cars on streets but not a lot has been done to actually race around a track.

What it does

The program uses deep learning to drive a car in Assetto Corsa, a racing simulation game. We used a Logitech G29 steering wheel to drive around the track for over an hour to collect data so we could train the network. It uses a supervised learning method.

How I built it

The program captures the screen output from the game, converts them to gray scale, and logs them into a file while simultaneously recording the position of the steering wheel, gas pedal, and throttle. We implemented our own neural network using Keras, a neural network API built on Tensorflow, to train a neural network to predict where the position of the steering, throttle, and brake should be.

Challenges I ran into

After training the network to make predictions we were not able to send the predictions back into the game to actually drive the car. Some ideas we had were to use a joystick emulator and send it the 3 values it needs to control the car but we were not able to find anything that has this capability nor create it in the time allotted.

Accomplishments that I'm proud of

Implementing our own convolutional neural network.

What I learned

Deep learning, logging data from screen capture and input controls, and the state of self driving car research (the standard for research is to use an open source game called "TORCS" which has horrible physics and is not a good place to train aggressive self driving agents).

What's next for Self Driving Car Simulator with Assetto Corsa

Once we can take the neural network's output and use it to control the car in the game we should be able to find out if our current system works well. From here we can begin to refine the algorithm. Once we have the car driving fast we want to incorporate reinforcement learning so the car can progress beyond our skill level to drive it around the track.