Interactive 3D Object Gallery

Using computer vision, our project gives users the ability to "interact" with any object in a virtual space, like an old statue or toy.

Comment

Inspiration

After hearing the Blue Heart group talk about sharing art with the community, we decided to make a more interactive way of presenting art that is harder to appreciate from just a 2D screen. We decided to model the controls after how Tony Stark would engineer products in the Marvel Movies.

What it does

Gives users the ability to rotate and analyze a collection of 3D and 2D models in different virtual settings using their hands, keyboard, or a bluetooth remote.

Foreseeable Uses

Advertising- able to see products in 3 Dimensions and see the inside of products ex. Cars and interiors Medical field- able to analyze 3D bone/cell structures, x-rays, CAT/MRI scans with immense details and ability to zoom Instructional- Teaching in multiple dimensions allows people of all ages to be able to see and interact while keeping learning interesting 2D and 3D modeling can be send over emails and is accessible for many people. 3D scans can be sent with iphone lidar sensors or 3D scanners as well.

How we built it

The visuals were all built in Unity with C# scripts controlling game logic. Two Python scripts were used to receive input from a joystick we built. The first code is used to connect Bluetooth and display the information provided by the remote on the screen. The second code handles relaying information from the controller to the computer screen interface.

Price

Any laptop or computer with a webcam has access to the free software (Unity with python backend calling Google’s free ML framework - mediapipe) Bluetooth controller costs $9-10 excluding external battery

We built a custom remote design in SolidWorks using CAD with a PETG Heat resistant material that houses the ESP32 Bluetooth Connector, LCD, breadboard, spring, micro usb connector, and Joystick. This allows us to be able to control the program from far away from the screen and allows us to have a more unique and interactive experience.

Challenges we ran into

For hardware, we ran into several wiring issues while dealing with connectivity from the voltage differences and requirements. On the software side, we ran into issues with having multiple hands show up, which interferes with the user's consistency in using the hand tracking.

While designing the Remote the measurements needed to be exact to mount the various circuit boards and grounds neatly and safely. Also 3D printing took several hours with a very small amount of time and took several variations and versions which required us to test fitment in the middle of the print to save time and material instead of waiting 3 hours for each variant.

Accomplishments that we're proud of

We have a functioning remote used as a source to change and deal with 3d models and artwork alike. which can be used in situations where you need to show models or artwork in a presentation setting. As well as tailoring the wiring to fit inside the given remote perfectly.

What we learned

We learned a lot about the complexity of how in-depth hardware and software connectivity goes. Dealing with issues and solutions that involve both software and hardware. Having 4 people with different skillsets helped us learn how to work together and make a joint product.

What's next for Interactive 3D Object Gallery

For hardware, we intend to optimize for size on the controller, as well as consistency, leaving fewer issues on the software side. We also wanted to add more options for input, giving more options for interface interaction on the remote.

For software, we intend to solve issues with connectivity and being able to track multiple hands at once. as well as trying to find innovative ways to use the limited amount of input options we were given.

Built With

Share this project:

Updates