Team Logo
GIF
Painting tutorial for roses
GIF
Painting tutorial for a dog

🎨📸 van.Go

Unleash the Artist Within! Now, YOU Can Be Your Own Van Gogh, Translating Your Words into Masterful Strokes with Cutting-Edge Computer Vision. Imagine, Create, and Let van.Go Guide Your Palette to Perfection!

🌠 Our Vision

Navigating through a creative block can be tough, especially for new and intermediate artists. Oftentimes the block is associated with turning an idea into actual strokes on a canvas often presents a challenging leap. For centuries, art has always been a powerful way to tell stories and share emotions tied to our memories. van.Go, inspired by the timeless art of Vincent Van Gogh and the latest advancements in AI and computer vision, is our tribute to this tradition. It's not just about preserving memories through beautiful art but also about making the complex world of artistic creation easily accessible to you, guiding your ideas into vibrant visual stories with ease and flair. In contrast to other AI art models, we seek to assist the artist, rather than replace the artist. van.Go bridges the gap between human creativity and AI generated art. The vision, final interpretation and implementation is at the will of the artist.

🌈 What it does

How can beginner artists convert their ideas and memories into art pieces on their own? Introducing van.Go. Users are able to enter a prompt to generate a painting and then overlay step-by-step visual instructions on their canvas. Providing a richer, multimodal interaction Bob Ross-like tutorial. Position your camera, choose your words, and let van.Go weave a tapestry of step-by-step visual instructions overlayed on your canvas, turning your every word into a vibrant spectacle of art. An artist resides within each of us, and van.Go brings that artist to life!

🔨 How we built it

van.Go was built using React, Node.js, Flask, Feed Forward Neural Networks, Style Transfer, OpenAI Dall-E, Open AI GPT-4, and a ton of caffeine. The platform integrates a frontend component for users like beginner painters and a backend to handle the user data, image generation, painting creation stroke-by-stroke, canvas detection, and many other components. We utilize the popular OpenAI APIs to generate high-quality images, which are then fed into Style Transfer, a neural network which produces realistic painting styles as prompted by the user, and then Painting Feed Forward Neural Networks to create the painting stroke steps. Lastly, the paintings are overlayed on the user live-video with ___.

🏔 Challenges we ran into

There were many problems that arose as we tried to implement an entire application in 36 hours. Our challenges can be categorized into 2 main areas.

Frontend & backend integration: API: Our project utilizes various APIs from OpenAI and many Github repositories. Managing that all of these calls to external code were consistent with our project was very challenging. Integrating all the different programs that we wrote for each different function of the application with Flask and Postman also posed a lot of issues because of data not being transferred between the front-end and back-end initially. Front-end design: Making a compelling UI is always challenging. Making a UI for van.Go was even harder as we tried to integrate artistic elements that took inspiration from our namesake Van Gogh and the Hack GT Journal of Memories theme. We also faced challenges on detecting the user's canvas. As a result, there were artifacts while placing the painting reference.
Model direction: After countless hours of research, we had at our hand the individual pieces of the puzzle. We had Neural Painting Transformers, which would create the brushstrokes, Dall-E which would generate the painting concept, and Stylized Neural Painting, which would transfer the style of one painting to another. However, merging these models together to create a coherent and aesthetic result was difficult. We painstakingly experimented to find the optimal conditions, performed a lot of prompt-engineering and retrained the models to produce the vibrant and stylized results in our demo.

It was through teamwork and effective communication between each of our team-members that we were able to stay motivated through this large-scale full stack project.

🎉 Accomplishments that we're proud of

UI: Our user interface featured artistic elements and color schemes that bound us to the Journal of Memories theme and our namesake Vincent Van Gogh. One of the most important features was our ability to overlay each painting step on top of the user's canvas which would make the process of following the tutorial much easier. We made a UI that any beginner artist could navigate and start their journey in making beautiful artworks.
End-to-End Painting Generation: Our entire back-end painting generation pipeline combines Dall-E, GPT-4, Stylized Neural Painting and Neural Painting Transformers to generate visually stunning paintings with brushstrokes. They are able to take reference images from the user and prompts from the user to help the user achieve their vision. We painstakingly integrated their best parts and after countless hours of experimentation, we were able to find their optimal conditions.
Teamwork: We were able to divide our entire application into 3 segments, the painting generation model, the user interface, and the communication between the UI and the model. This allowed us to develop each part simultaneously which saved us a lot of time in creating a project of this scale in 30 hours. We also portrayed strong communication which allowed us to ensure that every team member had clarity on the direction of the project. This allowed us to put up a very coherent project.

🎓 What we learned

Integrating Back-end and Front-end: One of the most important things that we learned was that we should've integrated the API calls between the back-end and front-end simultaneously rather than waiting until the end when each part was ready independently. We also learned that the design decisions on both front-end and back-end should be aware of how the API calls would be made to reduce the amount of code needed to be rewritten.
Front-end development: We learned principles of effect UI design by taking feedback from our peers on our application and looking at references of other applications. We researched color palettes, logo design and asset creation to create a consistent application.
Model Experimentation: We learned how to prompt engineering Dall-E and GPT-4 to create paintings that would work well with our Neural Painting model. We experimented with Canny Edge detection to ensure that detail is not lost between the Neural Painting model and the reference image. We also learned the trade-off between generation time and a better looking output when choosing our final model.

🔮 What's next for van.Go

Short Term

Richer AI Models: During our research, we found models that did a better job of adapting the user reference images to given art styles. However, due to the high computation cost and generation time, we were unable to integrate them into our project. If we deploy our web-app on a server, we would be able to use these richer models to make better looking style aware artworks.
AR: We faced a lot of challenges in using OpenCV to overlay the painting tutorial onto the user's canvas in varied lighting conditions. Using AR APIs such as MagicLeap, we would be able to leverage their deep learning models to better detect and overlay the painting steps onto the user interface.

Long Term

XR: Mixed reality is the logical next step for van.Go. Leveraging the recent advancements in VR and mixed reality such as the Apple Vision Pro and Oculus Quest, we can integrate the van.Go to create 3D artworks that involve your surroundings to enhance the way you relive and visualize your memories.