Inspiration
Have you ever tried to summarize a large, complex financial document with traditional AI and been unsatisfied with the results? Maybe it doesn't cover enough, or it hallucinates. Well, we have a solution for you!
What it does
RAGs2Riches is a MultiModal RAG application which allows users to upload any file they have especially financial reports, and receive answers to complex queries through our LLM. Simply upload your file, start asking questions, and let the magic flow!
How we built it
RAGs2Riches isn't your typical RAG app. We first extract all the text from the uploaded pdf. We then send it over to two separate retrieval functions, namely Dense Retrieval (using vector similarity in FAISS), and Sparse Retrieval (using keyword-based matching), and then put those 2 together into one Ensemble Retrieval which combines the strength of both, complimenting each other to be an even more accurate retrieval technique. Dense Retrieval utilizes bi-encoding by using a sentence transformer to encode both the query and each chunk into vectors, and evaluating the semantic similarity of those vectors using a cosine similarity score. Meanwhile, Sparse Retrieval looks for keywords from the query within the chunks. By having the ensemble retriever output the top k most relevant chunks rather than simply the most relevant, we also enable our model to find multiple pieces of relevant information scattered throughout the page that together answer the user’s query. Enabling the model to accurately process follow-up questions involved using Langchain’s question-answering chain for tracking conversation history. We prompt-engineered our LLM that gave outputs to, if necessary, rewrite the user’s query in order to capture all necessary context before passing it into our Ensemble RAG retriever.
Challenges we ran into
Early on in the challenge, we wanted to also include G-Retriever, a graph-based question-answering system built by LeCunn et al. in order to take advantage of its ability to reduce hallucinations. However, we had trouble getting Rebel to convert the pdf text into valid entities and relationships. Re-Ranking (explained in the Next Steps section) also proved to be a challenge because when we get docs from both retrievers, some of them had overlapping IDs, so when we tried creating a temporary retriever from FAISS for re-ranking, there was an error due to the duplicate IDs.
Next Steps
Incorporating G-Retriever and re-ranking algorithm into our app is what's next.
Log in or sign up for Devpost to join the conversation.