Yapply | Devpost

Inspiration

Drawn to the hackathon's theme word of 'Boost' and the social / education track, we started spit-balling ideas on what to make. During our brainstorming session, one of our teammates stepped out to take a call from his parents, who spoke another language. That's when it occurred to us... all 3 of us had family that spoke foreign languages, and we all shared the same struggle: a lack of conversational fluency.

The biggest aspect of a lot of modern day language learning apps is learning through memorization and retention, through flashcards, spelling, etc. But not one focuses on the most practical - and challenging - part of learning a new language that often gets overlooked: speaking it! Holding a conversation in another language is a whole new ball game than just reading and writing in the language.

We decided for this year's SASE Hackathon, we would build a competitive language-learning app centered around speaking rather than memorization. Think you know more Spanish than your friend? Hop in a Yapply room together, and put your skills to the test.

What it does

Yapply turns language practice into a fast, two-minute speaking game. Two players join a live voice room with a share code, get a speaking challenge, and talk. When time is up, each player gets a score broken down into conversation, fluency, pronunciation, grammar, and vocabulary, and specific things to improve, such as a grammar correction or a mispronounced word. Grading looks at how well you actually communicated, not just whether you used the right words, so practice feels like playing a game and not doing a lesson.

How we built it

Next.js, React and TypeScript power the frontend and API. Calls run on LiveKit with room-scoped tokens, and the server keeps the match timer so both players see the same clock. The two players talk directly, and analysis runs alongside the call so it never adds lag.

In the browser, an AudioWorklet captures only the local microphone and cuts speech into short clips. The server sends each clip to Azure Speech pronunciation assessment, which returns accuracy and fluency.

A separate Node worker scores each finished match with one Google Gemini call, so both players are judged by the same rubric. Gemini returns validated JSON for grammar, vocabulary and conversation. We combine it with Azure's pronunciation and fluency using the weights from our design doc.

Tools used to build Yapply: Next.js · React · TypeScript · LiveKit · WebRTC · Azure Speech · Google Gemini · PostgreSQL · Supabase · Node.js

Challenges we ran into

The LLM grader was the biggest challenge of this project, as it was the main crux on providing analytics for a user's progress and ranking their language skills.

Accomplishments that we're proud of

We're proud of being able to deliver and ship a functioning prototype. The data analytics aspect of our project was the most exciting to integrate, with Azure Speech transcribing and analyzing the words with Gemini providing useful insights.

What we learned

We learned how to use Azure Speech to assess pronunciation and provide feedback on spoken language. We also gained experience with LiveKit to connect two players through real-time audio and video.

What's next for Yapply

Leaderboard System: We plan to incorporate a leaderboard system that'll motivate our users to push towards their goals of improving their language proficiency.

Badges and Streaks: We plan to create a badges and streak system so we can to reward our users for using Yapply and improving their profiency in the language of their choosing.

More Language Support: Yapply started with English and Spanish. Our schema and the scoring pipeline are already language-aware, and seven more languages are set up in the database, including French, German, Japanese, and Korean. Turning each one on is mostly a matter of testing pronunciation scoring and tuning the grading rubric for it.