Dialectosoar

Inspiration

Some of our peers enjoy Bollywood movies and Spanish songs but don't speak the language. When talking to them, we learned that they are interested in learning the language without going through the traditional, repetitive process of reading, grammar, and listening. We used this as inspiration to make an interactive language learning tool available for a variety of languages and topics (recipes, songs, educational videos, demos, etc).

While there are tools such as Google Translate, this fails to account for the very nuanced and contextual nature of how language is communicated, and these tools are more for immediate translation needs rather than full-on language learning. While there are even language learning apps like Duolingo, we feel like these apps feel too academic and may fail to fully reflect the culture and the constantly changing quality of languages.

Furthermore, it becomes a great challenge for many of our older generation family members who are not as familiar with a newer language and the varying dialects that naturally emerge, so we also had a strong focus on promoting diversity and inclusion by allowing for virtually every pair of languages there.

What it does

A user drops a link to a YouTube video with closed captions and selects the language the video is in and the language they want to learn. After fixed time intervals, questions appear on the screen asking about the video content in the time interval. Question types range from comprehension multiple choice to direct open-ended translation. These questions are randomly generated every time. After the video is over, words that you learned during the video appear on the screen, building a personal dictionary.

How we built it

We used a YouTube API for embedding videos into the site when you paste in a link, as well as automatically pausing and replaying the video between questions. A separate YouTube captions API was used to extract the captions of all the languages and the timestamp of those captions. For generating questions as well as vocabulary term extraction, we harnessed the powerful OpenAI GPT-4 model. Finally, for storing learned vocabulary after watching a video, we used Firebase's Firestore to seamlessly store user data persistently across learning sessions.

Challenges we ran into

First, it was tough to figure out a good API and data source to use for the purposes that we wanted. A challenge specifically with the OpenAI API was ensuring that our prompts were generalized enough to produce a well-structured output for any video out there so that it would match with the UI format for questions and storing our data. Another design challenge was making the site seamless enough to make the learning experience enjoyable. Of course, so many people want to learn languages, but sometimes you might not feel like sitting in front of a textbook and feel like you're in school. Nicely combining both entertainment and education was our mission.

It was difficult to decide the fixed time interval after which the video would stop because different videos are at different speeds, so we had to closely look at the character count during the intervals to decide when to stop and ask a new question with new content.

Accomplishments that we're proud of

We started with a few languages our team knew and expanded to a variety of foreign languages. Randomly generated questions add value to our product because it expands the topics that can be tested. Finally, we successfully included open-ended questions such as text translation.

What we learned

Half of our team was first-time hackers, it was a great experience for them to learn react, front-end tools, and deal with API calls on the fly. The vibe too of hackathons!

What's next for Dialectosoar

We hope to develop a feature that will allow users to continue practicing based on their performance and skill level. After completing a video lesson, the questions bar will be replaced with recommended videos so learners can take their skills to the next level.

We would also add more personalization features such as creating an account on the website so progress is saved, score reports can be generated based on performance, and catering to the learner's goals.

Finally, we would like to scale this so that it can work with videos lacking closed captions and become a product other companies such as Amazon and Netflix could use for their entertainment platforms.

Furthermore, the vocabulary lists could be taken to the next level by exporting to other flashcard services such as Quizlet or Anki.

Built With

• api
• express.js
• firebase
• openai
• react
• youtube