VinA: AI-powered visual novel generator

Logo for VinA, an AI visual novel generator
Sample base character generated using Stable Diffusion
Sample facial expressions using txt2img masking
Sample background image of an art studio
Sample background image of a park

Inspiration ✨

Our inspiration stems from the desire for anyone to create an inclusive, customizable story. Whether they wish to bring their LGBTQ+ story to life, explore a fantasy world with new characters, or dive into an alternate sci-fi reality—we wanted to build that dream for them. Our team is currently working from two different time zones, with one member in Windsor, Canada, and two others studying abroad in Japan, so we wanted to create a visual novel story that anyone can use from anywhere. We were also interested in integrating AI storytelling within the project to push the limits of what we could create; this led to VinA.

What it does 🎉

VinA asks the user for a story description in a CLI to generate a visual novel complete with multiple characters, different facial expressions, dialogue, varying backgrounds, and ambient music. Then, the user can interact with this RenPy visual novel and enjoy the new music and graphics that enhance the original story. The same story description can also generate a slightly different visual novel with every usage. The novel is then automatically run.

How we built it 💻

We built VinA in Rust, using Stable Diffusion API for character generation, facial expressions, and background images and ChatGPT API to create Stable Diffusion prompts, dialogue, and a script. Afterwards, we compiled and modified the API responses in Rust to work with the required RenPy formatting for scene descriptions, music, and character placement.

Challenges we ran into 🛠️

The main challenge was working with unfamiliar technology like the Stable Diffusion and ChatGPT API. There was a learning curve to determine the best parameters (e.g. Inference Steps, Guidance Scale, and Scheduler, which helped with the number of denoising steps, following prompt information, and the denoising process, respectively). Furthermore, we needed to specify the internal ChatGPT prompts to create the best prompts for image generation based on character and background descriptions.

There were also issues we ran into when putting the entire project together; we each worked on different parts like Stable Diffusion API calls, ChatGPT API calls, or RenPy game generation, so these files in Rust needed to build on each other to get the final visual novel.

Finally, there were problems with speed since generating images and text with AI both take significant amounts of time. We tried to speed up the process by caching information and limiting the number of calls to both APIs.

Accomplishments that we're proud of 😊

Our goal of having a CLI for a visual novel generation based on any given prompt has been accomplished. In addition, we are proud to have used Stable Diffusion and ChatGPT effectively to enhance the final product and are amazed at what these technologies could accomplish. We overcame our difficulties by consulting each other and debugging while also being able to pick up tasks and work together seamlessly.

What we learned 🧠

Since we had never used Stable Diffusion or ChatGPT API before, it was great for us to learn how to integrate these APIs in our final project. Furthermore, we learned about the importance of caching and refactoring code, especially since the difference was measured in minutes rather than seconds for image generation. Rust was also interesting to use to create the API calls and modify the API responses to satisfy the requirements for RenPy.

What's next for VinA 🌈

We hope to create a more user-friendly interface and further decrease the time it takes to generate a visual novel so everyone's stories can be told.