ImmVRse

AI Generated Avatars for the Metaverse.

Comment

Inspiration

Being students, each from different backgrounds, we have always understood the importance of connecting with people different from us. We see the Metaverse as an opportunity to accelerate these connections, and understand the necessity of making these connections feel as real as possible. Many of the current Metaverses rely on animated avatars, which don't account for real life features, both physical and psychological. Additionally, we felt the Solana community lacked any real virtual reality on-ramp for on-boarding users into Web3 and the ecosystem itself.

What it does

ImmVRse enables users to create AI generated, hyper-realistic, avatars to use in virtual reality. These avatars are stored on the Solana chain as NFTs and are intended for use as avatars in Solana metaverses.

How we built it

We used the PIFuHD model (credit: facebook) and BackgroundRemover (credit to the creator of https://backgroundremover.app for all code related to that as well as the API itself) in order to create a system where images can be turned into masks of the people inside of them, and these masks (along with the initial images) can be used to generate a 3D model of the person. This model was then fed into MeshLab, where a series of scripts (not included in the Github) were used to convert the formats outputted by the PIFuHD model into a format that could be read by Unity, our choice for a VR development tool. We then fed these models into Unity, and used Web3.storage (a filecoin project) to store all these assets on the IPFS storage network. Finally, we plugged the CID's into various document formats (PDF's and PNG's for 2D views of the model) and turned these into a NFT through the Solana CLI. We then did this process for all four of us to create an NFT Metaverse avatar for ourselves in our own phantom wallets.

Challenges we ran into

We had extreme difficulty utilizing Unity Software. To begin with, we encountered stiff resistance when the output of our 3D model would change from colored to black and white. We figured this out by writing a number of scripts in mesh flow and translating between the encodings of OBJ Vertex-Based Coloring to OBJ/MTL/Mask based coloring. Additionally, we ran into a number of issues with rendering Virtual Reality components in Unity and with converting a series of 3D objects into a continuous stream of Virtual Reality content. However, we got around this issue by purposefully rate-limiting the frames/second of our Unity VR rendering and creating a custom per-frame update function.

Accomplishments that we're proud of

  1. The whole team worked on new technologies.
  2. We combined a great deal of different technologies, often to great success.
  3. We integrated realistic 3D models into virtual reality.
  4. We created a system that could potentially be turned into a VR live-streaming application, and could be used to generate a more "real" metaverse.

What we learned

This project lies at the intersection of many technologies - Machine Learning and Generative Modelling, the Solana Blockchain and NFT's, and the world of virtual reality (that we can hopefully make a bit more "real"). Working with all of this tech allowed our team to learn a lot about the intersection between Web3 and AI and the world of graphics. We learned about the Solana blockchain, deployed our first NFT's on-chain, and worked with decentralized storage systems (IPFS) to store our avatar files. We learned about the cutting-edge in dimensionality-increasing machine learning models. We learned about developing virtual reality-applications on Unity. But most importantly, we learned about working together effectively as a team, dividing development (and design) tasks efficiently between people, and allowing each person to play to their strengths while also getting to learn about a new piece of technology.

What's next for ImmVRse

ImmVRse will first continue developing and working towards on-boarding individual users. Long term, we want to provide services to creatives and content creators so that they can stream to their audiences worldwide. This will help both the artists and the Solana community by integrating new fans into the ecosystem. In addition, by creating an incredibly easy and realistic-looking (a rare trait in the Virtual Reality world) on-ramp into the Metaverse (on Solana!) we can grow the Solana ecosystem and the Virtual Reality aspects inside it at unprecedented rates. Solana is an ecosystem devoted to speed, reliability, and ease-of-access. We bring that to the Metaverse inside of it.

Built With

Share this project:

Updates