extractED | Devpost

Inspiration

Have you ever just finished a lengthy lecture, and you're staring at a dense powerpoint presentation filled with complex concepts and crucial information? Or, are you studying material for upcoming exams, but the thought of synthesizing multiple resources to manually create notes and study aids seems daunting, time-consuming, and inefficient?

We have faced this situation all the time. We find ourselves struggling to learn content from our classes the day before our exam, and wishing we had a more efficient way of filtering through and studying the content we actually needed to learn.

Our application, extractED, acts as a powerful tool to intelligently generate concise notes, divided into intuitive sections, and create personalized quizzes based on the material, complete with an auto-grader that provides instant feedback on your understanding—allowing you to take charge of your learning like never before.

What it does

First and foremost, our application leverages various natural language processing techniques to parse through lengthy PowerPoints and identify various topics and subtopics within the PowerPoint. We then provide a concise, textual summary of each topic broken down into bullet points for easily accessible and readable notes for each powerpoint. Then, our autonomous AI agent automatically creates and grades a quiz generated on the notes for every PowerPoint.

We use natural language understanding models to interpret and evaluate the correctness of the answers on the quiz based on semantic understanding of the generated notes, and direct every user to the sections of the notes that cover topics missed on their quiz. We also provide each user with a personalized Youtube playlist of videos that cover their missed topics.

Finally, our application also includes a fine-tuned ChatBot, meticulously trained using data relevant to the topic, designed to respond to particular user inquiries with supporting references while preserving the history of the conversation.

For example, a college student taking a history class, would simply upload their history PowerPoints to our web application. Our web application would then parse the powerpoint, intelligently generate notes sectioned off by each topic, and then generate a corresponding quiz to test the student on the material. After the student takes the quiz, our AI agent would identify the topics that the student missed, direct them to the corresponding sections in the notes and provide additional helpful YouTube videos to help them learn more about the topic.

Target audience: Students—to create a more efficient and personalized learning experience

How we built it

This project can be broken down into the following steps:

1) Parsing the Powerpoint We iterate through each slide in the slideshow, and identify “text” objects using the python-pptx library. We use Python’s NLTK (Natural Language Toolkit) to tokenize the text and split it into sentences.

2) Generating Notes We utilize the Hugging Face “all-MiniLM-L6-v2”, and create latent embeddings for each sentence generated in the previous step and then compare each sentence using cosine similarity on a similarity matrix graph. These help us identify “clusters” which then serve as a basis for our topics and subtopics within our notes. Then we pass each cluster of sentences to Google Gemini and have it name each cluster, which then serves as our topic header.

We divide the text from each cluster using a CharacterTextSplitter to generate smaller documents. These are then processed in a map-reduce style for summarization, employing a LangChain summarization chain. This approach enhances the capacity to preserve a wider context while distributing the workload.

We then pass these topic headers and summaries into Google Gemini and used prompt engineering to generate concise, yet descriptive bullet point notes.

3) Generating and Grading Quiz We use prompt engineering to provide Google Gemini with topic headers and the corresponding notes to generate a multiple choice quiz, specifying how many questions we want per topic.

Leveraging Fetch.ai’s framework, we created an AI agent that automatically grades the quizzes after the user takes them. We define a “quiz” model and grader agent that includes a “grading” protocol. The grader agent has a grade quiz method that takes a “quiz” object and grades the quiz by comparing the user's responses with the correct answers.

This agent also takes in the notes that were generated in a previous step. We use a CharacterTextSplitter to split the notes under each topic header. We use OpenAI to create latent embeddings for these notes, and store them in a FAISS database. We take the list of missed questions (ones with incorrect answers), and for each missed question, we perform a similarity search on the FAISS database using Langchain to map the most relevant note document and then map the note document back to its corresponding header.

This allows us to return the specific topic sections associated with every single missed question. We pass in this list of topic sections to the Youtube API to generate a list of Youtube Tutorials for further learning on each topic.

4) Fine Tuned Chatbot Leveraging the Langchain framework, we first ingest relevant data and segment it, followed by the transformation of text into latent embeddings using OpenAI Embeddings, and storing these embeddings in a vector store. We also maintain a conversation history for each user. This history is dynamically incorporated into the chatbot with each interaction. Simultaneously, the system retrieves pertinent information swiftly from the vector store. The resulting prompts are fed into an OpenAI Large Language Model (LLM) to facilitate informed interactions with the user.

5) Frontend and Flask Our frontend almost entirely uses the React.js framework assisted by Chakra UI. We use a Flask server to connect our front end and back end.

Languages: Python, Javascript, HTML, CSS

Frameworks: React.js, Flask, LangChain, OpenAI, Google Gemini, Fetch.ai, Chakra UI, HuggingFace, YouTube API

Challenges we ran into

API rate limits—OpenAI rate-limits required a more efficient processing mechanism and fewer round trip requests for embeddings.
Building an AI agent—This was our first time using Fetch.ai so we had to learn about different concepts such as Mailbox, Protocol, Context, as well as how protocols and agents communicate
Random sentences—When testing sentences without much content and a lot of filler words, it was hard to come up with a “cluster” grouping and name and it would make the graph we were using a lot messier
Accuracy—Coming up with algorithms to improve the accuracy of our natural language processing techniques.

Accomplishments that we're proud of

We are excited that within 36 hours we were able to build an end-to-end product that utilizes various NLP techniques and leverages LLMs to create a more efficient and intelligent learning experience for students. We loved the idea of building something that we would enjoy using regularly as students ourselves. We were also happy that we were able to use so many open source models to build something that has the potential to be valuable to so many students in the education space.

What we learned

Throughout this project, our whole team became more familiar with how powerful various technologies and frameworks are. For many of our teammates, it was our first time using Fetch.ai agents and leveraging Langchain and Hugging Face models, and it was really cool to see how useful autonomous agents, semantic understanding, and even text generation APIs could be.

This project was a really educational journey for our team, as we tackled several technical challenges to build something we all believed we would personally use.

What's next for extractED

Use a batching algorithm to simultaneously run multiple AI agents to make it more scalable
Integrate using the Canvas LMS API to directly run our program without having to upload powerpoints
Enabling automated fine-tuning of our chatbot based on the notes generated the powerpoints the user uploads
Live lecture note generation so students could use this software during class
Quiz based off of previously missed questions