82 - brownie | Devpost

Scanned model
Point map rendered in real time on live camera feed
Generated heat map

Inspiration

Our first idea was to come up with a system to aid stress and heat testing for mechanical engineering. When we were discussing the plan, one of our teammates jokingly mentioned that this system could be applied in the kitchen as well, "perhaps to bake a brownie". The name stuck!

What it does

Using the TrueDepth camera found in the iPhone X and newer models, the Brownie mobile app allows you to scan and create a 3D point cloud with no extra hardware. The point cloud is sent to a web server hosting the backend code, which generates a heat map of the object. This heatmap is finally sent to the 8th wall, allowing you to view the heatmap in real-time utilizing AR. The point map can also be saved locally to your device as a .gltf (3D model) or .ply (point cloud) file in case you want to access them.

How we built it

We implemented each portion of the app independently, utilizing 5 different languages (Swift, Python, Javascript, CSS, and HTML) Once each component was completed, we wrote software to interface between Swift in the frontend app and HTML in the backend webserver.

Challenges we ran into

We hit obstacles almost every step of the way. We had the most trouble with the 8th wall library, due to our lack of experience with the platform and finding it difficult to use with confusing documentation. We also had to struggle with XCode issues and ended up consuming 30GB of hotspot data to fix these.

Accomplishments that we're proud of

We were able to access the TrueDepth camera and create a fully functioning 3D scanner. We were also able to generate a heatmap for 3D models utilizing differential equations.

What we learned

While talking to other groups and discussing among ourselves we realized that our project could be used for many other things such as helping firefighters calculate the damage cause by fires and also helping engineers do stress and heat testing.

What's next for Brownie

We want to continue working on 8th wall and making it better; in particular we would like to add concrete timescale, and temperature measurements. We also want to try and streamline the process for generating heat maps, and overlaying. In addition it would be nice for the overlay to be partially see through something that we did not implement.

Built With

Submitted to

Los Altos Hacks VII

Created by

I wrote frontend and created the IOS application, along with the 3D scanning system for the app. I also wrote code to interface between swift and the webserver based backend.

Shaurya Sood
I worked on portions of the python backend, linking the python backend to the swift frontend, and worked with the 8th wall library for AR.

Ansh-123 Thakkar
I worked on researching on how to use 8th wall and overlaying images using it. I also made the entire pitch and did marketing.

Aaditya Mathur
I worked on back-end, mostly on the finite-difference differential equation solver. This was to generate the heat map overlay.

Rishi Sridhar