Inspiration
Reddit's r/place is a collaborative pixel art canvas where users can place colored pixels, creating a dynamic and evolving communal artwork. We wondered generative AI tech in 2023 could reinvent collaborative multiplayer art projects like these.
What it does
FigGPT is a ChatGPT plugin that connects users to a global canvas. They can describe in natural language how they'd like to modify on the canvas (e.g.. "Add a blue square under the tree") and see the changes reflected on the a public webpage in real time.
Additionally, users can stylize the canvas using natural language prompts like "make this a watercolor painting" that completely revamp the style of the canvas while maintaining its structure.
How we built it
We developed a GPT plugin that connects to the global rendering server. Users build and submit prompts via the ChatGPT plugin, which add or retrieve + modify portions of the SVG canvas.
The stylization feature was built using Stable Diffusion + ControlNet conditioned on the rasterized canvas.
Challenges we ran into
- Maintaining canvas state when users refer to different items - we had to build a 2D retrieval system over the canvas space
- Ensuring GPT-4 outputs valid SVG.
- Generating interesting images with Stable Diffusion that maintain high alignment with the canvas. Our final model even works on text (e.g. "Add the phrase hello world to the canvas", "now light it on fire and make burning text")
Accomplishments that we're proud of
We're excited about the potential of AI-enhanced collaborative work in the creative sphere, and FigGPT is a promising step in that direction.
What's next for FigGPT
We plan to expand FigGPT with a richer object language and fine-tune its details, making it an even more powerful tool for collaborative creativity.
Just like r/place, we're excited to see the art emerges and evolves as more people connect and play with the canvas over time.
Built With
- gpt-4
- openai
- python
Log in or sign up for Devpost to join the conversation.