Abcent | Devpost

Inspiration

Growing up, we saw the struggle our immigrant parents faced with speaking English. They often had a hard time not only with accents but also with common American speech patterns and slang. English-speaking ability is also heavily related with perception in the workplace and public, as well as with freedom of expression. As a result, we decided to work on this project to help other immigrants like our parents develop a better accent and the confidence to speak English in regular conversation.

What it does

Many immigrants grew up learning English from classic TV shows and movies. Abcent helps its users learn how to speak with a native accent by playing clips from popular American media along with a transcript of what the characters in the clips are saying. After playing the clip through, the user will then be able to practice their own pronunciation by saying what the characters were saying, receiving a confidence score and feedback indicating how well they pronounced each word. This way, the user will be able to identify where they can improve and further practice.

How I built it

We used iOS frontend written in Swift with a Firebase database and Python cloud functions for the backend. We used the Google Cloud Speech API for speech recognition and used metrics of individual word confidence and cosine similarity with word vectors for the pronounciation feedback.

Challenges I ran into

One of the biggest challenges we ran into was audio transcoding. Apple and Google use very different audio encoders, and we had to experiment with our own settings in order to create truly smooth interactions between the two.

Accomplishments that I'm proud of

We were proud that we could effectively use the API to develop this project in the span of the event. By showing teamwork and efficiency, we were able to split the tasks and get each item done systematically.

What I Learned

What we learned through this event was that sometimes in can take a while to get started, not only with just finding an idea but also with figuring out where to start in terms of building the project. We also learned about how careful analysis of the documentation of APIs can help speed up the process, since then we get a better understanding of the product instead of just copying it.

What's next for ABCENT

Currently, we only have clips from an episode of "Friends" loaded into the app. In the future, we hope to have a much wider range media sources, such as other TV shows, movies, radio shows, or interviews. In addition to this, we would also add accounts to the app so that the user could keep track of their progress and see how well they do and their improvements over time