Never Lay Down | Devpost

In this demo, a Trump simulator impersonated by AI where you get to have a full conversation by speech with him, answered with an AI voice that also impersonates him, all of this in virtual reality. We used 3 AI models (ChatGPT, Whisper, FakeYouTTS).

Inspiration

We were inspired by wanting to create AI characters in video games. We wanted to test out how it could be done with Open AI models and Unity.

What it does

The user can talk with any character, and AI will replicate a persona. For our demo, the user can talk to Donald Trump by speech. Open AI Whisper's model recognizes and transcribes the speech to text. This prompt is then sent to our back end that queries ChatGPT. It returns a text, a facial expression and a body movement. The text is then sent to a text to speech model, FakeYouTTS, that impersonates Trump's voice, and sends back a URL to the audio file. Unity thus receives a JSON of these information, where it downloads the audio file and the body movement associated will make Trump move while he speaks (the audio file plays). The user can continue to converse with Donald Trump was long as they wish.

How it works

Unity send a request of the user's speech input to the back end that does API calls to openAI models such as ChatGPT or models such as FakeYouTTS. Wee use Whisper model to get the speech of the user, and transcribe it to text.

Challenges

We ran into challenges with the animation of the character. We had to make sure that animations looped and transitioned correctly.

What are we proud of

Impersonates perfectly with animation, text to speech, speech to text. VR is included as initially planned. 3 AI Models!

What we learned

How to make AI characters with unity

What's next for Never Lay Down

We want to incorporate more characters.

Built With

c#
chatgpt
fakeyoutts
fastapi
mixamo
openai
python
unity
whisper

Submitted to

McGill CodeJam 13
- Winner Committee's Choice

Created by

I worked on processing the request from unity and send them to apis, and process response data to make them available for unity.

KILLERWHOPPER 皇堡
I integrated VR functionality and Speech to Text using the Whisper model. I also worked on the communication between Unity and the backend by sending requests and receiving responses to handle audio and animation processing.

Alison Quach
I worked on handling the backend API integration, ensuring the seamless communication between Unity and the AI models (ChatGPT, Whisper, FakeYouTTS). My primary focus was on processing requests from Unity, making API calls, and handling response data, including audio files and animations. I also worked on optimizing data flow to ensure the responses were received and processed efficiently in real-time, allowing for smooth conversational interactions and animated responses from the AI character.

Jiaduo Xing
I imported the character model and designed its animations on Mixamo. I also wrote the code to choose the animations order based on the AI output and to make them trigger only when the character is talking.

TCholette