Goose Painter | Devpost

Inspiration

We were inspired by the concept of skribbl.io, a popular online multiplayer game where players have to draw their given words and other players take turns to guess them to get points. We wanted to know if AI fares better at this sort of image recognition than humans, thus we made Goose Painter!

What it does

Goose Painter works by getting an LLM to judge how well you can draw to describe a theme, with a twist. For a given theme, each player will be able to draw three images that describe the theme and its characteristics. Once completed, the app will generate a keyword or short phrase describing what it perceives in the images. With these keywords, the app then generates a short story building around them. Finally, each player's drawings are judged by comparing the generated story of each player against the original theme phrase using a defined rubric. The player with the highest similarity score is deemed the winner of the game!

How we built it

We built our application on the React framework, using JavaScript for our website and FastAPI for our endpoints. We leveraged GroqCloud and its API to access their vast array of LLMs to generate the keyword descriptions of each player's drawings, which is then sent back to our backend using the FastAPI endpoints. To generate the unique stories for each player's drawings, we send the drawings into LLAVA model via Groq which gives us the key words for the app, which then sends the keyword descriptions to Cohere's Command R LLM. Finally, we also use Command R to generate a story that communicates the drawings. And then those drawings are sent into Command R where we do a similarity scoring to evalute how close it fits to the initial. The rubric that we fed Command R follows for a criteria based on relevancy, tone, flow and creativity. Each player's drawings are ranked based on this score, and the player with the highest score is crowned the winner!

Challenges we ran into

We ran into a lot of issues as we developed Goose Painter, especially with the backend side. We initially had difficulty with integrating the Groq API within our code, as well as stitching the drawing keyword descriptions into a single input. However, we were able to overcome these challenges through perseverance and the help of our wonderful mentors!

Accomplishments that we're proud of

We are really proud to have set up our sophisticated backend correclty, as we have to ensure that each API call is being done in a sequential manner. We are also really proud of our UI and website design, which provides users with a fun way to get their drawings in!

What we learned

We learned a lot on how to leverage APIs and LLMs, especially across various clould partners and bringing them all other in sync under one application.

What's next for Goose Painter

We want to make it a more customizable experience where users can set certain themes, change the LLMs prompting, choose how many pictures they can draw, and overall ways to enhance the user's ability be more creative via our tool. Also, we want to implement this system into other similar games such as Dress to Impress such that there wouldn't be human biases in games that require judgement. Additionally, we want to explore new ways and tools to improve the accuracy and robustness of the similarity score generation.