Inspiration
Since filing taxes is such a tedious process, we decided to create a real time API which acts like the user's personal accountant in order to help filing taxes, understand tax documents as well as provide key data analytics based on documents provided.
What it does
It allows a user to upload documents and consult the real time API to fill tax documents and understand them too. The voice supported Multi Agentic Real Time assistant allows the user to ask questions regarding the documents and also see data insights/analytics based on documents supported which are generated in real time. Also provides anomaly and fraud detection services based on info provided
How we built it
We have a real-time agent powered by a WebSocket that streams data between the Twilio API and our real-time API. Built on a RAG system, the agent retrieves precise information to answer user queries and leverages multiple tools to identify and complete the correct forms based on user needs. Using PyPDF Filler, it extracts fields and fills forms from verbal input and ingested documents in real time, simulating a human-like interaction. Completed forms are then emailed to users for signing and filing. Uploaded documents are processed through OCR, extracting and summarizing key data, which is stored modularly in MongoDB for efficient retrieval. Our fine-tuned model generates insights, recommendations, and dynamic visualizations based on data type and volume. For fraud detection, we apply Benford’s Law to compare natural data distributions with user documents, assigning a risk score based on deviations. Additionally, anomaly detection flags inconsistencies by analyzing extracted data, comparing similar fields, and highlighting irregularities.
Challenges we ran into
We ran into challenges related to providing data analytics and anomalies. This is because the user can attach any number/type of documents for the voice assistant to process. There is no fixed schema for the documents to upload hence everything is happening in real time
Accomplishments that we're proud of
We are proud of interweaving Twilio along with Websockets and the OpenAI real time API in order to provide a virtual voice who we can call in order to process documents and fill the forms.
What we learned
We learned about providing real time data analytics and also more about real time API.
What's next for TaxDaddy
We aim to create a system which can directly upload documents and file taxes without any human intervention apart from approving form filing.
Log in or sign up for Devpost to join the conversation.