Text to VR

Inspiration

VR is expensive to create. Requires many resources to provide quality content. We want to create an affordable way to bring VR to people.

We created a unique never-before-seen Text to Virtual Reality project.

This project would be great for hospitals that want to provide patients with custom VR content without spending a lot of money.

This project can also provide calm spaces for health care professionals who want to de-stress without having to spend a lot of money.

What it does

If you've ever wanted to explore a different world without hiring a whole gaming department to build it, then this project is for you! This project can be used by patients who want to relax in a virtual reality world by just typing a few words. It can also be used by patients who want to explore a different world without being able to leave their hospital room.

How we built it

We started by using clipstyler which is an Image Style Transfer with Text Condition. We realized that it would generate usable images but they weren't realistic enough. We later found Stable Diffusion which enabled us to initialize the AI model with an image and convert it with a prompt.

Challenges we ran into

We ran into a problem when creating these images as they weren't VR ready. They were just images. We researched lots of technologies including Neural Radiance Fields (NeRFs) and settled on generating tile-able images. This enabled us to build out environments without using too much computing power. We can now generate VR scenes for about 10 cents.

​​

Accomplishments that we're proud of

To construct a 360 scene, we also experimented with AI image generators such as Night Cafe, which utilizes Stable Diffusion to build any view from imagination. By providing reference photos of a scene from different angles (e.g. hill on a sunny day) and a style reference (e.g. mars landscape), we were able to generate a series of photos that are consistent enough to composite a scenery resembling what we had imagined.

Another viable solution was using a high-res panorama photo and combining it with a style reference photo.

What we learned

Besides building whimsical scenery, VR also offers a new way to experience art, and possibly a new form of art therapy. Imagine creating a scene that simulates the scenes from a classic painting. With this technology, you could walk into any famous artwork, exploring what inspired great artists to create these masterpieces.

The purpose of this project is to create a simple VR experience with a low level of programming from the user's end. Ultimately, we want to strip the barrier between advanced technology and those who are not tech savvy but still like to enjoy the unlimited possibilities VR could provide.

What's next for Text to VR

We want to make it easier to build libraries for VR content. For example we want to bring in-painting directly in an environment. So someone could use their controllers to draw over areas and speak-to-text to create new objects.

Built With

• instant-ngp
• nerf
• python
• stablediffusion