ModelVision | Devpost

Inspiration

The world of education is rapidly changing, but many traditional learning resources remain static. Textbooks with 2D images and text often fail to engage students or help them grasp complex concepts effectively. Inspired by the possibilities of Augmented Reality (AR) and the potential to transform traditional learning tools, we envisioned a solution that brings these static resources to life. Our goal is to create an immersive, interactive learning experience that can make complex topics engaging and easier to understand.

What it does

ModelVision transforms textbook images and diagrams into interactive 3D models that students can explore using their smartphones or AR glasses. With just a scan of an image, users can see detailed AR visualizations that enhance learning by making abstract or complex topics more tangible. The platform also integrates an AI-based assistant, allowing students to ask questions about the models and receive contextual answers to aid their learning.

How we built it

We used Unity for developing the AR functionality, integrating Vuforia's AR SDK for image recognition. The 3D models were designed and implemented using Blender and imported into Unity. For the AI conversational assistant, we leveraged Google Cloud’s AI services, which provided robust machine learning models for natural language processing. These models were integrated into the AR environment to offer context-aware explanations.

Challenges we ran into

One major challenge was optimizing the performance of AR models for different devices. We had to balance the detail and quality of the models while ensuring smooth performance across various smartphone configurations. Another challenge was integrating Google Cloud’s AI services into the AR environment in a way that allowed for seamless interaction between users and the 3D models.

Accomplishments that we're proud of

We’re proud to have developed a working prototype that effectively bridges the gap between traditional education and modern, immersive technology. Successfully integrating both AR and AI in a way that enhances learning is a huge accomplishment. We also made significant strides in optimizing performance, ensuring that users could experience smooth AR visuals without sacrificing model quality.

What we learned

We gained in-depth knowledge about the AR development process, particularly about optimizing 3D models and making AR systems device-agnostic. We also learned how to combine two cutting-edge technologies, AR and AI, in a meaningful way. Our experience highlighted the importance of user experience in educational tools, and we gained a deeper understanding of how students interact with learning technologies.

What's next for ModelVision

In the future, we aim to expand ModelVision by enabling textbook publishers to create specialized AR models tailored to their content, allowing for a more personalized and interactive learning experience. We also plan to incorporate student tracking features to monitor progress and reward learning through gamification, making the educational process more engaging and motivating for students. Additionally, we will focus on continuous overall improvements, enhancing both the performance and usability of the platform to ensure a seamless and enriching user experience.