AI Quest | Devpost

Inspiration

We wanted users to be able to create and chat with a customized AI character, since manually gathering data to create a character is tedious and boring. We also wanted to experiment with artificial intelligence, auto-data-collection, and more.

What it does

It hosts a customizable AI chatbot that you can have conversation with—through both text and speech—on a locally-hosted website, with auto-generated personalities and backstory.

How we built it

We built the front-end with HTML, CSS3, React, and Next.js, which took text and audio prompts and displayed output from our back-end. The back-end was built using llama_index(LLM framework), dataloaders that get information from Wikipedia, retrieval augmented generation(For enhancing character performance), other techniques such as ReAct Prompting and Chain-of-Thought Prompting , Google Gemini Pro, Whisper (Speech to Text AI), and ElevenLabs (Text to Speech AI), which processed the text/audio from front-end and returned the character's AI-generated message as text/audio. The communication between front-end and back-end was facilitated using Fast API, by hosting a local server that listens to POST requests from the front-end.

Challenges we ran into

We had to learn how to operate new frameworks and integrate RAG into our program. We tried to work with live2d and Computer Vision, and more. We also found the front-end to be very difficult to program, as well as connecting the front-end and the back-end, since all of us didn't have experience in Fast API, but we managed to implement both.

Accomplishments that we're proud of

Being able to implement most of the aforementioned libraries and systems, and have them work together to produce a functional model. We also managed to complete a significant portion of the goals we initially had at the beginning of the project.