-
-
The app in my Zoom Apps
-
The app running natively in Zoom
-
Generated feedback email sent to student after tutoring
Inspiration
While attending the MBA program at the University of San Francisco, Valerio Mirarchi (Graduating May ‘24) has been working as Language Tutor, teaching Italian to undergrads at USF and adults at Istituto Italiano Scuola in North Beach, San Francisco. Through these 2 experiences, he has identified numerous insights about individual learning opportunities lost after online language tutoring sessions.
What it does
Duets.ai “learn language smarter” - serves AI-powered smart summaries and personalized learning tools that helps people, who pay to study languages through online meeting sessions with tutors (market segment), avoid losing individual learning opportunities (problem).
A plug-in solution, integrated with Video Conference software, leveraging AI and LLMs can answer this need by providing highly personalized learning tools, opportunities, and content out of every single tutoring session, helping users to get the most out of them and learn effectively. That's what Duets.ai does.
How it came to be
During Cal Hacks 2023, Valerio introduced the idea to Pooja and Parsa. They decided to team up with him, and the 3 started collaborating on a prototype that could capture some of the key features of the original idea.
How we built it
We worked simultaneously on creating the backend ChatGPT API service as well as the front-end Zoom interface. The Zoom app was built starting from Zoom’s official quick start example, using Pug/Jade and Node. The backend was built using the OpenAI quick start example for text-completion. It was started as a stand-alone notebook and then modified to a Flask app for integration with the front-end. The mail-service was built after we had the server hooked up — we used flask-mail for that.
Challenges we ran into
Originally we intended to transcribe a Zoom meeting between the language tutor (predominantly speaking the language to be learned) and student (mostly answering in their native language) to then directly feed into GPT to generate handouts and practice exercises. However, since there would be a constant switch between the two languages during a tutoring session, we could not simultaneously transcribe the meeting in both languages. That’s why, for the purpose of an achievable demonstration during the Hackathon, we opted for using feedback input fields for recording the session. We did, however, come up with various potential future solutions involving various tagging systems and redundant transcription but noted that tackling that would be a whole project on its own.
Another issue we faced was trying to send automated emails via SMTP, only to find out that the protocol was restricted on the venue network (sigh) and that took a lot of debugging until we eventually resorted to using our personal hotspots.
Further, making our code work with the quick start examples led to a lot of module conflicts and unexpected errors, but thankfully we had ChatGPT on our side ;)
Finally, drafting the appropriate prompt for the ChatGPT API required a lot of trial and error as it was highly unpredictable initially. After playing around a little with the temperatures and penalty settings we finally figured the correct configuration tailored to our use-case. Another challenge with using the API was that the output would not format nicely. For us to draft the HTML email we needed it to respond in a valid stringified JSON format, however the AI was rarely predictable and raised lots of edge cases; we tackled them as best we could. For instance, we resolved invalid JSON outputs (missing quotes, extra characters, etc.) by feeding the original response back into another ChatGPT prompt asking it to return a valid JSON output.
Accomplishments that we're proud of
On the engineering side, we were fairly unfamiliar with hosting an API on Flask but together, and with a lot of trial and error, we managed to get something up and running that met our initial MVP goals for this Hackathon. We also learned about prompt engineering and the value of having the right prompt for the job. Finally, we managed to create a semi-working Zoom native app despite the severe lack of public documentation. Even though, the native app mostly works as a proof of concept for future implementation for now, we are glad to know that it is largely just due to the API restrictions and our time constraint.
Overall, we are extremely proud of being able to:
- Team up in a nice and propositive collaboration spirit
- Positively share thoughts about the use cases identified and suggested by Valerio
- Productively brainstorm around challenges and come up with potential solutions
- Define and follow an achievable plan for the ~2 days
- Better understand some of the current limitations of ChatGPT and its capabilities
- Release a working prototype
What we learned
On the engineering side, we learned how to build a Zoom app, work with OpenAI API and automate the process of sending emails via an SMTP protocol. Moreover, we worked on prompt engineering in a different light to maintain consistency.
We all also feel like we learned how to collaborate better amongst a cross-functional team.
What's next for Duets.ai
The next step in Duets.ai would be to be able to transcribe the meeting and feed it directly to ChatGPT. This would save the tutor's efforts to add feedback comments, and we could directly generate insights, exercises and other highly personalized content from the session (which is highly interactive and unstructured by nature) directly.
We want to further develop the solution and evolve it into a platform to be offered directly to consumers with a subscription model. We have many innovative solutions in mind and are excited to take this to the next level!
Log in or sign up for Devpost to join the conversation.