ChatRPG | Devpost

Inspiration

Wanting to have a novel gaming experience where we can interact with characters and the environment in a more free and dynamic way

What it does

Prompts ChatGPT to play the role of a DM, taking its outputs and generating images from them, generating text to speech from the responses, and (half finished) generating music to match any shifts in mood.

How we built it

In python using ChatGpt API for text and reasoning, DALLE API for images, OpenAI text-to-speech API for audio narration, and Suno AI API for music generation. We also use llama_index and milvus for indexing vectors for character info.

Challenges we ran into

The many different components, sub-agents calling chatgpt, maintaining relevant context, and updating the vector database became very complex for the scope of this project, so we had to strike a balance for an MVP.

Accomplishments that we're proud of

Every piece of content is AI generated! The only thing that isn't novel each time is the title screen/intro, and music, but we will soon be adding novel music as well.

What we learned

A lot about prompt engineering, how to use different chat agents for different sub-roles, how vector databases and queries work, how to run local open-source AI models (future ideation)

What's next for DungeonMaster

More robust world and NPC management (using individual vector stores per character), more consistent scene generation that honors the character appearances, music generation, hosting this on the cloud and running our own open source AI models for every component, automatic world generation (user can ask to role play in any world and it will generate vectorizable assets for that world)