Language Lens

Our project utilizes real-time computer vision to help users visually learn everyday items in their chosen language, whether it be for learning a new language or for expanding their current vocabulary

Comment

Inspiration

Learning a language is always difficult, but we aim to ease this process by turning the real world into an interactive, real-time classroom.

What it does

As you point around the room with a live camera, you get instant detection of everyday items, providing labels, translations, and simple definitions in real time. In addition, by simply tapping on an item, the user gets the correct text-to-speech pronunciation. of the chosen object.

How we built it

We built Language Lens by combining a set of different systems. More precisely, for object detection, we utilize the YOLO model to find objects and return coordinates and labels to the client. Furthermore, we connected a translation system and a text-to-speech model to serve data in different languages. To the user, this is added as an overlay in real-time on top of the video. Several optimizations, such as caching, were also used in order to speed up the pipeline.

Challenges we ran into

Finding a model that balances speed and precision was a main challenge because our needs expected real-time output while being accurate. Furthermore, having the application work on all systems was also a challenge because we all had different setups.

Accomplishments that we're proud of

We were able to provide a fully functional system in a short period of time in fields we were new to.

What we learned

We have learned how to build an AI-driven application by combining multiple models into one system, and more specifically, we gained experience working with computer vision and managing performance through optimizations and caching. We also learned the importance of UI design when presenting results in live interactive applications.

What's next for Language Lens

Scaling to provide more languages and optimizing further for even better object recognition. We also aim to go beyond individual items by creating relational sentences between 2 or more objects. Finally, we foresee integrating this type of technology into other devices such as smart glasses so users can turn their everyday environment into a language learning space.

Share this project:

Updates