What it does
The bot uses computer vision to distill and find the ball, and then simulates keystrokes on a user's laptop to automatically move the paddle
How I built it
The bot uses mss and grabs a portion of the user's screen. Then it finds all countours(intersecting areas of two different colours with different intensities) which aids it in finding the ball and the user's paddle. Once it has the x and y coordinates of those, it references the two and uses the user's keyboard commands to send the paddle to line up with the y coordinates of the ball.
Challenges I ran into
Because everything in the game is white on black, the algorithm had a tough time distinguishing between the objects it needed to track(ball, paddles) and the objects it didn't need to track(score numbers, divider lines). This made the paddle very unpredictable and sometimes undetectable because of the sheer amount of wrong things the script was tracking.
Accomplishments that I'm proud of
In order to get past this, I had to implement optical flow to track only the objects in motion. In order to filter down to just the ball, I had to write a series of parameters for each contour to pass in order to be registered as a ball. Parameters such as the contour's density and size of pixels, its position on the screen and the speed at which it was moving. After much trial and error, I made it so that the algorithm tracked only the objects it was supposed to and stored their y positions in a class to be referenced against one another.
What I learned
What I learned is that sometimes, you're not going to have anyone to help you but yourself. When I was struggling with the algroithm tracking the wrong objects, I looked for many alternatives online for detection. Everything from colour detection to haar cascades, but none of them solved my problem. I had to come up with the filtering routine myself and after hours of painful trial and error, it finally worked. There's not always going to be a tutorial or documentation to solve your problem, and being able to resolve issues on your own is the difference between a good hacker and a mediocre one.
What's next for Pong bot
In the future, I'd like to implement an online feature through the use of tools such as google firebase. In essence this would mean that when livestreamed, a script running on a remote device would be able to capture the game screen, run the same algorithm but instead of simulating user keytrokes, it would instead send directional commands to a firebase database which would then be read from by the device running the actual game. This would allow a user to control a game of PONG that's not actually running on their device and would be a massive step towards more interactivity for streaming sites like twitch and YT live.
Log in or sign up for Devpost to join the conversation.