PetAGI

• 

• 

• 

  Pull Request of all the code changes made in this hackathon

Inspiration

Some people can't afford to own a pet. They can have this virtual pet and be able to travel around the world.

The objective is to take care of a highly intelligent 3D Creature that you can talk to. It has a life span of 7 days. You have to feed it 2 times per day, and you have to play with it once per day. The interactions are in the Vision Pro. The 3D creature lives on your desk. You have to keep it’s energy and happiness levels up so that it grows and does not get sick and die. If you don’t feed it or play with it, it will get sick and die. If it get’s sick you can give it medicine and it will get better.

What it does

Objectives - key features: 1.⁠ ⁠voice interactions - chat with voice 2.⁠ ⁠⁠Animations - happy, disappointed, excited, dancing 3.⁠ ⁠⁠Game mechanics - feed and energy level goes up, play and happiness level goes up.

How we built it

We used GPT-4 for the Ai. NextJS for the backend with Mongodb Database. Reality Kit, Reality Composer and Mixamo for the 3D avatar and animations. Vercel for hosting. Daily for text to speech and speech to text.

Challenges we ran into

Ran into a problem with Daly, we posted a github issue: https://github.com/daily-co/daily-ai-sdk/issues/63

Accomplishments that we're proud of

The animations are playing based on the conversation. There is an LLM call to determine which of the 7 animations the avatar should play based on the conversation.

What we learned

ARKit is really complex, and it requires a lot of code to handle edge cases, hand gesture recognition, recognition of surfaces and more things to handle with code.

What's next for PetAGI

We really wanted to make the Pet appear on your desk, and for it to move around. This would make the effect of immersion and the feeling of realism of the creature much more impactful and engaging.

Built With

• mongodb
• openai
• swift