TourgEYEd

Innovating the tourism industry by introducing augmented reality and computer vision technology to give users a modern, informative, and accessible travel experience.

Comment

Inspiration

The awesome campuses of Berkeley (Go Bears!) and Stanford + a love for AR and computer vision shared by the both of us.

What it does

Early-stage prototype of a Microsoft HoloLens app that allows wearers to access information "tagged" onto notable buildings, art installations, historical landmarks, etc through CustomVisionAI and a holographic windowed display.

How we built it

After a full day's worth of brainstorming and research, we decided to complete a project that would be applicable to the real world while applying AR and computer vision tech. Previous experience working with Unity and the Microsoft HoloLens 1, .NET frameworks, and machine learning in conjunction with consultation from mentors from Magic Leap and Microsoft allowed us to perfect our ideas and our plan of action. The first step to making this project possible was training a model of various buildings and sculptures from around Stanford and Berkeley campuses, while also creating a basic mixed reality Unity project. Afterward, we started integrating the mixed reality platform with the Azure dataset/model, which took up the bulk of our time.

Challenges we ran into

Although we were successfully able to create such a model and Unity project, many of the Unity libraries containing valuable classes, methods, and information had been lost due to un-updated documentation from the Web.

Accomplishments that we're proud of

We had very high success rates with testing our CustomVisionAI model; even before training the AI with nighttime and rotated images, it was able to recognize the buildings and landmarks with very high (90% or higher) accuracy.

What we learned

We learned plentiful amounts of valuable skills toward utilizing Microsoft Azure, .NET frameworks, MRTK 2, and the nuances of handling multiple forms of multi-platform documentation.

What's next for TourgEYEd

We are planning to fully integrate the output of the Azure dataset, shown in JSON string form to be readable in a C# script for a Unity GameObject so that it can interact with the augmented reality scripts developed on the Unity side of the project. In addition, further planning has been made to develop accessibility tools for VIPs (Visually-Impaired Persons) with voice recognition software and text-to-speech guidance of tours.

Built With

Share this project:

Updates