KnowNonsense | Devpost

Inspiration

Thanks to recent media and political cycles becoming increasingly careless with spreading false information, fact-checking has become essential for all of us to remain unbiased. As a team, we quickly agreed that we could achieve something great despite widespread misinformation and unproductive debates. KnowNonsense is our take on uniquely approaching those moments when we need the truth to stay on topic.

What it does

KnowNonsense listens for points made by people during debate, and feeds back in real time on each point made if it's true or false and why. It displays that in virtual reality, then sends the entire transcript of the conversation with the corresponding true or false labeling and concise explanations neatly to the website. Where they can be interacted with all together.

How we built it

Headset records all the dialogue.
Each phrase/sentence is stored as a .wav file, that is sent as a filestream to our FastAPI backend.
Each dialogue is converted from speech to transcribed text.
In order to determine whether or not it should be fact checked, we perform some natural language processing on a fine-tuned model of the Hugging Face claim detection model to identify whether or not there is an objective claim or a statistical claim.
If so, it is sent through Google's LLaMA, which provides sources and an analysis of the facts.
These are saved as JSON, and sent back to the VR headset to display, as well as being saved as a file in Pinata, where:
- A PDF transcript of the discussion is created and saved for users to download.
- A view of the dialogue is provided.

Challenges we ran into

While developing a feature in which the debate would be written out, fact checked, and displayed, live in its entirety on the website KnowNonsense communication between FastAPI and the Front End became extremely convoluted, causing the program to fail or be far too slow.

This was also the team's first experience developing a VR application, which brought about some frustrating issues and complications. Switching from Unity to Unreal Studio for the VR scripting proved to be slightly less complicated.

Accomplishments that we're proud of

We are proud of how we overcame numerous obstacles in implementing some of our most crucial steps. By effectively combining inputs into our pipeline of machine learning models, we consistently achieved accurate and fast results every time.

What we learned

As we progressed through each step of development, adopting new technologies was essential to the project. Learning Unreal Engine presented a significant challenge due to its steep learning curve. Nevertheless, our workflow improved dramatically as the hardware, front-end, and back-end components learned to play their roles and communicate effectively. We even had a brand new team member who contributed to every aspect of the project but was particularly immersed in web development.

What's next for KnowNonsense

The next phase for KnowNonsense involves enhancing and expanding the virtual reality experience. We plan to integrate more interactive elements within the VR environment, allowing users to engage more deeply with real-time fact-checking feedback. Additionally, we aim to refine our machine learning models for even greater accuracy and speed, ensuring users receive instant and reliable information during debates. We're also looking to broaden the platform's accessibility by supporting more devices and incorporating augmented reality features.