speakAR | Devpost

Inspiration

Have you ever wished you could say a word and have the item simply appear in front of you?

We noticed that no such technology currently exists, but 3D generative AI is a growing space. Using this to our advantage, along with NLP and AR integration, we came up with speakAR.

speakAR is a state-of-the-art visualisation tool with infinite use-cases. speakAR will be leveraged by other developers for artistic expression, experimental ventures, and other practical scenarios. For example, its scope will be expanded to education as a visual supplement to verbal lectures or as a learning aid for neurodiverse learners. It will also be used in healthcare to visualize detailed anatomy of the human body. In short, 1 tech, endless possibilities.

What it does

speakAR transforms free-form human speech into 3D models that you can view in your real-time surroundings through augmented reality.

How we built it

Natural Language Processing (NLP): Converts complex speech (natural language) to text.
3D Generative AI: Uses MirageML, DreamFusion (diffusion model), PointE, and ShapeE to generate 3D models of the inputted speech and ensures conversion from text to 3D is accurate, smooth, and free of distortions.
Augmented Reality (AR) Integration: Overlays the generated 3D visuals onto the real world for an immersive experience using ARKit/ARCore.
Mobile Application: Cross-platform interactive application (Android and iOS) created using Flutter, Dart, XCode, and Android Studio.

Challenges we ran into

Cross-platform AR integration was challenging (having the app run on both iOS and Android seamlessly)
Learning to build from scratch since the team had no prior experience in AR development. We took it as a stride though and had fun while at it!

Accomplishments that we're proud of

We are proud to have a created a novel technology that has infinite applications in different fields like education, innovation, art, and communication. We were able to make this app cross-platform which increases its accessibility exponentially.

What we learned

We developed skills in Flutter and ARKit/ARCore for iOS/Android application development. We learnt to adapt and problem solve given tight deadlines.

What's next for speakAR

We hope to take this technology to the next level by breaking into the realm of XR/VR. We envision speakAR being open-source to be built upon for the aforementioned infinite possibilities too!