Inspiration
The inspiration behind our project stemmed from the desire to create a novel Augmented Reality (AR) experience by leveraging generative AI for 3D model generation.
What it does
Our application offers users a unique platform to generate and manipulate 3D models through text commands, providing an immersive AR experience. Users can craft personalized items, such as hats, and seamlessly integrate them into their surroundings, fostering interaction with the virtual objects.
How we built it
The foundation of our application was laid using a blend of technologies, including React, MediaPipe, Text-3D-Model, MindAR, MongoDB, Next.js, and A-Frame. Text-3D-model played a crucial role in the generation of objects based on user-provided captions. MediaPipe was employed for face tracking, facilitating the accurate mapping of 3D-generated objects onto the user's face. The rendering and interaction aspects were handled by MindAR with A-Frame. MongoDB served as our reliable database, while React and Next.js were instrumental in orchestrating the seamless integration of various components.
Challenges
Our journey was not without challenges. We encountered difficulties in achieving fluid controls, especially in integrating MediaPipe with face mesh. Ultimately, we decided to forego hand controls due to their clunky nature, prioritizing a smoother user experience.
Our biggest challenge was time, we had a lot of things we needed to implement, such as getting the face mesh to work, the whole pipeline to generate new objects, adding all the tools necessary for the user to interact with the generated objects, and using React to make all the components work together. We also used MongoDB for caching since the rate limits of our API use was really low, since we were originally querrying from the front-end but we only had 10 queries per page-load, so there was errors on page load, instead we hold the results in our database and query them for free.
We also had to use MongoDB to store images that people can take using our website, which makes it so what we don't have to deal with any storage bucket systems like S3.
Accomplishments
We take pride in the quality of engineering exhibited in our final build. Overcoming on-the-spot compatibility issues between different models showcased our problem-solving abilities and contributed to the overall success of the project.
What we learned
The learning curve was steep, and we gained valuable insights into the utilization of new models. Additionally, we discovered the intricacies and challenges associated with coordinating controls in virtual reality (VR).
What's next for Genesys
Future plans involve the incorporation of hand tracking technology and body segmentation. These enhancements will empower users to try on accessories and costumes in a more immersive manner, expanding the application's capabilities.
Log in or sign up for Devpost to join the conversation.