Tum Assistant | Devpost

Inspiration

Technische Universität München has a very cool student app, which can inform students about lessons, opening hours, the location of the nearest U-Bahn station and so on.

Unfortunately, oftentimes it is cumbersome to find this information, because there are a lot of menus and many features. Also, if you are in a hurry to reach the next lesson, or you are wearing gloves because the Bavarian winter is very cold, you cannot use the touch screen.

To combat this, we have chosen to make the interaction with the app more spontaneous and user friendly by creating an assistant, integrated with the official app, that uses speech recognition, Language Understanding Intelligent Service (LUIS), and text to voice technologies.

The TUM Campus app now sport a new “TUM Assistant”, that will be your best companion for your studies at TUM!

What it does

TUM Campus app has a lot of different implementations. We created the TUM Assistant plugin in a way, so that it will be easy to implement new features. We focused on training it on 3 existing services and creating a new one completely from scratch!

It is able to give you information about near public transportation: It uses the position of the phone to give you useful information, like 'When is the next train?', 'Where is the bus station?' or also 'When does the last subway leave?', so you can always know when you have to leave the university to do not miss your transport!

TUM Assistant is able to retrieve data about the nearest Mensa and inform you about its location or the menu today.

It has also access to the database with the information about all professors and students at TUM, so you can ask for the email of a fellow student or where to find your professor.

Other than that, we have implemented a new feature for Computer Science students, that is not available in the main app: you can (nicely) ask your assistant to print for you a file directly from the download folder of the phone!

And thanks to luis.ai, all these communications can be natural and you do not have to remember commands, just talk to him!

How we built it

The core of the app is luis.ai, a Microsoft technology to add language understanding to your application: we have trained it to recognize different intents, one for each feature we have implemented.

We have then created the assistant inside TUM Campus app to query luis.ai and categorize his replies. Thanks to this, it will be easy to add new features: after a question by an user has been categorized, an API is called to provide the right answer to the user.

We have also used Android’s specific service to make interaction easier: the assistant is integrated with Google Now, so you can query it without even touching the screen, and you can listen to the reply instead of reading it thanks to the text to speech feature!

We focused on Android because it is the only MobileOS that the TUM Campus App supports, but since the core of our project is luis.ai, it will be easy to extend it to other systems as well.

Challenges we ran into

We had to train luis.ai to recognize a lot of different phrases, sometimes very similar, to understand the right intent.

Accomplishments that we're proud of

We were able to create a full working system you can use just with your voice, and you do not need to touch the screen, thanks to the “Ok Google” Feature.

Also, it is able to recognize lots of different inputs, and it will be easy to expand to support new features as well.

We are especially proud of the printing function, which will make using the university printers as easy as talking someone. (Don’t even have to touch your phone!)

What we learned

How to train a Language Understanding Intelligent Service to understand different sentences in different contexts.

Also, we improved our knowledge of Android APIs, like Google Now integration and text to speech service.

What's next for Tum Assistant

We are looking forward to make all the TUM Campus App features available through the assistant and to integrate it into the public app, so every TUM student can use it.