[G14] - InkSense

Immersive Language Learning

Inspiration

Our inspiration for creating this app stemmed from the desire to make language learning more engaging and practical. Traditional methods often lack real-world context and can feel disconnected from daily life. We wanted to harness the immersive power of Mixed Reality (MR) to create a learning experience that directly connects language with the user's environment. By incorporating handwriting recognition users build muscle memory —a powerful cognitive tool for retaining information.

What it does

InkSense is a Mixed Reality language learning app that combines real-world object recognition with AI-powered handwritten text recognition. When users start the app, it identifies objects in their environment, such as chairs, tables, and other items. Users then write the names of these objects in the target language using a high-precision stylus, controllers, or hand gestures. The app's AI analyzes the handwriting and compares it to the correct object name, providing immediate feedback through visual and auditory cues. If the input is correct, the object is highlighted, and a sound confirms success; if incorrect, the app guides the user toward the correct spelling.

How we built it

We built InkSense using Unity as the development platform, integrating Meta's Presence Platform for an immersive Mixed Reality experience. The passthrough and scene understanding features were essential for accurately identifying real-world objects within the user's environment. To achieve precise handwriting recognition, we utilized Unity Sentis, which allowed us to incorporate AI/ML models for real-time analysis locally on the device. Additionally, we integrated the Logitech MX Ink, a high-precision stylus, to offer users a seamless handwriting input experience in MR.

Challenges we ran into

One of the biggest challenges we faced was ensuring that the handwriting recognition was both efficient and precise. The Mixed Reality environment presented unique difficulties in maintaining the accuracy of handwriting inputs, especially when accounting for variations in user handwriting styles.

Accomplishments that we're proud of

We are particularly proud of developing a unique real-time handwriting recognition system within a Mixed Reality environment. The handwriting recognition is performed locally on the device, ensuring that all data is processed offline, which significantly enhances user privacy and data security. The app's spatial object identification feature ensures that the labeled text is accurately positioned in proximity to the actual object, creating a highly immersive and realistic learning experience. This precise integration of handwriting recognition with spatial awareness and privacy-focused design sets our app apart and offers a truly innovative approach to language learning.

What we learned

Throughout the development of InkSense, we gained valuable insights into the complexities of combining Mixed Reality (MR) technology, spatial awareness, and object recognition with AI-driven handwriting recognition. We explored the intersection of MR and AI, understanding how these technologies can be harmonized to create next-generation applications. Our journey highlighted the potential of MR and AI to transform traditional educational tools into dynamic, interactive experiences.

What's next for InkSense

The next steps for InkSense include adding support for multiple languages to broaden its global appeal. The current version of the app focuses on real-world objects, we intend to introduce virtual objects into the user’s environment, offering more diverse and customizable learning scenarios.

We aim to integrate voice detection and text-to-speech generation. This will allow users to practice speaking and listening, rounding out the learning process.

we are exploring ways to make InkSense more gamified, introducing elements such as challenges, rewards, and progress tracking to enhance user engagement and motivation.

With promising updates to Meta's Presence Platform, such as SceneScript. A novel AI-driven approach for 3D scene reconstruction that enhances semantic segmentation and expands object categories. We plan to significantly improve object recognition in InkSense. This will allow the app to identify a wider range of real-world and arbitrary objects, making the language learning experience even more immersive and precise.