CompanionAI

It's basically the smartest and coolest pet in the world. It can talk to you, teach you things, and do what you want it to, aimed particularly for kids who struggle with speaking.

Comment

Inspiration

We wanted to create something that solves a problem in the world but not just any problem - one that needs our skills to solve. We came together as software and electrical engineers to come up with Companion AI.

What it does

It's basically a pet that talks to you. It's aimed at helping young children with speaking problems get more comfortable with having conversations. It can also be used for children who have English as a second language. Moreover, it can become a friend to children who struggle to make friends. It's also very knowledgeable and can act as a sort of elementary teacher to a child and answer all the children's curious questions, nurturing their development and resulting in a smarter generation of people.

How we built it

We used React on the Frontend and Python on the backend to simplify things. We had some help from a half-done Mechanical Engineering project leftovers as the main frame and we implemented the code ourselves. We have a frontend that authenticates a user using auth0 and records an audio file. The audio file is then processed server-side(via Gemini API speech-to-text and prompting) and sent to a Supabase database, from where it is fetched by a Raspberry Pi who is continuously looking for updates. Upon receiving a prompt, in case of an action, it completes the action through PIGPIO and in either case uses ElevenLabs API to generate a voice that responds to the owner, essentially creating an illusion of a smart pet.

Challenges we ran into

I can go on and on about this. The biggest challenge was getting the hardware working as none of us had any experience whatsoever with a hardware hack. Integrating an already complicated AI-heavy project with that was a pain. Not to mention, we had a lot of difficulties combining the moderate frontend with the heavy backend and making flask and react work seamlessly together to make a user-friendly application that also works as an AI-powered remote control to the Pi-powered robot. There were also severe hardware limitations as we learned that we had a driver that accepts 5V input and a Raspberry PI that outputs 3.3V, hence needing a logic level shifter. The problem? It was 1 am and we just couldn't find any. The mentors were also past their time and although they tried their very best, it was somewhat too late. Due to our hardware implementation failure, we decided to simulate the the robot's actions through using LED lights as markers: RED: going forward BLUE: going backward YELLOW: going left GREEN: going right

Accomplishments that we're proud of

We aimed high this time, got out of our comfort zone, and said "Even if we can't finish it, we'll try to make something super impressive. If we fail, so be it!" Of course, we somewhat failed as a lot of features couldn't be implemented due to both time and hardware restrictions but we architected a proof of concept and left room for development. It has become a project we'll continue to work on beyond the hackathon, only this time collaborating with Mechanical Engineers to work out the logistics of movement better. It will become an actual pet.

What we learned

We learned exactly how early we should start thinking about the hardware. A lot more things would have been implemented if we didn't realize the need for hardware at 1 am. We also learned a lot about robotics as it's our first time doing anything remotely similar. We learned about different processes to implement the same thing such as websockets or database fetching - there are differences in difficulty, speed, and accuracy. We learned to use different tools like ElevenLabs and PIGPIO, generally opening all our worlds to a whole new set of tools out there. It was a truly eye-opening experience.

What's next for AI Robot Pet

The possibilities are endless. Our main target that we unfortunately could not finish implementing (we started it) is a computer vision feature where it uses facial recognition to recognize its owner. It will always try to tilt its head(a webcam or something like that) towards its owner, make noises, and greet its owner. Then we can have it following its owners. We were also thinking of repurposing these to help the elderly and people with physical disabilities get done with regular everyday housework.

Built With

Share this project:

Updates