-
-
connect email
-
Upload Audio
-
Transcript
-
summary
-
Summary Email
-
Raw Transcription email
Inspiration
The inspiration for this app stems from me frequently missing meetings, I thought I could find a way to get the key parts of a meeting even when I missed it . This app provides a solution for those who often miss meetings by offering a way to get transcriptions and summaries of the meetings they missed.
What it does
The application allows users to: Connect an email address Upload an audio file of their meeting. The audio is then transcribed into text. The raw transcription is sent to the user's email. A summarized version of the transcription is generated. The summarized transcription is also sent to the user's email. Both the raw transcription and the summary can be viewed in the app's UI and can be downloaded by the user.
How I built it
The app is composed of two parts: The frontend: The frontend is built using React, a popular frontend JavaScript framework. It leverages the @nylas/nylas-react package to handle user authentication and email sending. Audio files are uploaded using the @bytescale/upload-widget-react package and a string is returned which is our uploaded audio. The string is sent to a backend server endpoint (/transcribe-audio) for transcription. A function summarizeMeeting is used to generate a concise summary of the transcription. The summarization process uses OpenAI's GPT-3.5-turbo model. The function summarizeMeeting: Creates a concise abstract paragraph of the transcription. Identifies and lists the main points discussed in the text. Detects any tasks, assignments, or actions mentioned in the transcription. Analyzes the sentiment of the text. All these summaries are then combined into a formatted string that is returned to the user.The app employs React's state and effect hooks (useState and useEffect) to manage user data, loading states, and to respond to changes in the app's state.
The backend: Sets up and manages the Nylas SDK for email functionalities. Establishes an open webhook tunnel to listen for events from Nylas, like when a message is created or an account is connected. Provides endpoints to generate authentication URLs, exchange authorization codes for access tokens, send emails, transcribe audio, and more. Utilizes a mock database (mockDb) for demo purposes, where user details and tokens are stored and retrieved. Transcribes audio using the Replicate library, specifically leveraging the 'openai/whisper' model.
Challenges I ran into
Major Compatibility Challenge: Initially, the project was built using Next.js. However, due to compatibility issues between the Nylas React package and Next.js, the project had to be abandoned and switched to a standard React setup. This shift required significant effort on my part Managing user authentication, especially when integrating third-party services like Nylas. Handling potential errors or unexpected inputs in the various endpoints. Managing OpenAI API costs and rate limits, especially when processing lengthy transcriptions.
Accomplishments that we're proud of
Seamless integration of audio upload, transcription, summarization, and email sending. Efficient error handling and user notifications using toast notifications. The ability to not only view transcriptions and summaries on the platform but also to download them. Advanced and comprehensive summarization using OpenAI, which provides not just a general summary, but also key points, action items, and sentiment analysis. Efficient integration of various services like Nylas and Replicate to provide a seamless user experience. Building a robust backend that supports both email operations and audio transcription. Implementing user authentication and session management effectively. Providing real-time feedback and updates using webhooks.
What we learned
Framework Compatibility: The importance of verifying third-party package compatibility with chosen frameworks (in this case, the incompatibility between the Nylas React package and Next.js) early in the development process to avoid significant overhauls Nylas Integration: Learned about the intricacies of integrating Nylas into web applications, especially when it comes to authentication and email functionalities. Nylas Webhooks: Gained a deeper understanding of how Nylas webhooks function, providing real-time updates and feedback on specific triggers, such as when a message is created or an account is connected. Backend development using Express.js and integrating various third-party SDKs. Handling user authentication and managing tokens securely. Using the Replicate library for transcription tasks, specifically leveraging the 'openai/whisper' model. Utilizing OpenAI's GPT-3.5-turbo model for advanced text summarization and sentiment analysis.
What's next for MailMeMinutes
Incorporating a voice recording feature directly in the app, allowing users to record meetings in real-time. Integrating with calendar apps to automatically send transcriptions and summaries for scheduled meetings.
Log in or sign up for Devpost to join the conversation.