Pantheon Meets

Ditch Zoom, with PantheonMeets we break the barriers of communication

Comment

Pantheon Meets is a project oriented towards technology and accessibility built by high schoolers starting from April 20th.

Inspiration

Advaith Narayanan, one of our team members, has a grandfather who hard of hearing. Advaith is pained to not be able to communicate properly with him. With this project, we hope to break these barriers in communication and bring those who are hard of hearing into a more immersive experience in the world.

What it does

PantheonMEETS is an online meeting service in the form of a website which allows users to join public meetings and connect with others in real time. Some features include toggling camera and microphone, screensharing, opening a public whiteboard, and captions. A captions box is present on all meetings which will automatically detect speech and convert it to text. Deaf users have the ability to communicate by using the chat box to type in the captions or by holding up American Sign Language(ASL) symbols, which the program will detect and transcribe.

How we built it

Our meeting website uses Agora RTC and RTM for direct communication. We used Vercel and Node JS for token generation, backend functionalities (routing) and hosting on https://meets.pantheonlabs.org/. We also use MediaPipe Vision for in-site hand detection. We also created an ASL phrase classifier using our own phrasal epenthesis algorithm that we created—current industry standard for movement epenthesis mostly require acceleration data, but we came up with our own method of doing so—utilizing machine learning to prune unecessary motions.

Challenges we ran into

We ended up needing to switch hosting platforms from cloudflare to vercel to get this to work. Our ASL phrase classifier took thousands of lines of python just to generate the data and train the model, which was an intense and difficult process, but was well worth it. Connecting the backend and the frontend also was painful

Accomplishments that we're proud of

We are very proud of getting the meeting website to work. We're also proud of being able to create the phrase classifier, which and we created a unique and almost breakthrough algorithm for predictions.

What we learned

This project was the first time we hosted a website, and used vercel for hosting. We learned many things about website hosting, and about client/socket/API connections.

What's next for PantheonMEETS

We are planning on refining this and showing this to our schools for them to use it.

Built With

Share this project:

Updates