StuDine | Devpost

Inspiration

For a lot students, deciding on which dining hall to go to for each meal could be a difficult decision with lack of information about the food and overall dining situation(like crowdedness, seating availability, and waiting time). Some dining locations (for instance, Faculty House) doesn’t have pictures in their menu and some stations (like the grain bowl station at John Jay) does not have a menu at all. Therefore, we were inspired to create a platform where students can post and read photos comments about the dining hall food and dining experiences that will make this decision easier. Plus, dining halls can be a great place to meet new people so we also wanted to bring the community closer with this application!

What it does

StuDine provide users with customized recommendations with their favorite food! After user submits a few photo comments on food and dining halls they like or dislike, the app use machine learning to recommend the best available food for the user at the current time. The food recommendation engine also learns on the edge without storing personal preference on our backend dataset to protect user's privacy. For example, you have liked the lemon cake, the cheesecake at Ferris, then the engine knows that you are a big dessert fan at Ferris. The next time Ferris serves chocolate cake, it will be recommended to you! More importantly, as an application that's made by students for students, users are able to share and receive real-time photo comments that are from similar perspectives about dining hall food and their experiences. With the unique friend function of StuDine, we want to provide not only a platform for sharing food reviews but also a place for people to meet new friends. Strong bonds are formed from common like or dislike on dining hall foods :)

How we built it

Design: We used Quasar and Visual Studio Code to design the interface of the web-based application.

Computer vision: We built a custom image classifier that we will train on the fly in the browser using TensorFlow.js, a powerful and flexible machine learning library for Javascript. First, we load and run a popular pre-trained model called MobileNet for image classification in the browser. We then use a technique called "transfer learning", which bootstraps our training with the pre-trained MobileNet model and customizes it to train for the application.

RNN recommendation: Our deep learning approach is inspired by recent text generation techniques that use RNN models capable of generating subsequent words from a given phrase (sequence of words). For our recommendation problem, we replaced sequence of words with sequence of items and applied the same methodology. Based on food classification results, we connect users’ attitude with each food category (through their ratings using the heart button). Then, we use Keras to preprocess data. We also added a self-attention layer which has been identified as a performance booster for sequence-to-sequence model architectures, as it is in our case. Model hyper-parameters have been tuned using the Hyperband optimization algorithm implemented in the keras-tuner library, which allows one to make distributable hyper-parameter optimization for TensorFlow models in few lines of code. Before going live with our new recommender system, we adopted a backtesting strategy in order to evaluate the model predicting performance.

Firebase: Our Vue.js backend connects to our Firebase which is running on Google Cloud. We mainly rely on Firestore as our relational database of choice due to its focus on scalability, parallelism and realtime updates. We also use Firebase to implement user authentication with native email/password method.

Challenges we ran into

When first designing the interface of the application, we faced a challenge of making the comment function to encourage user submissions. From personal experiences in other apps, we found that having a separate comment page that doesn't have a proper route that lead the user to the page will make users less likely to upload comments. So we utilized a camera/photo-taking path to lead users to the comment submission page where the photos they take will be automatically part of the comment. Another challenge we faced at the early stage of building was how to give recommendations based on comments. We found a solution by classifying user uploaded images with their attitudes. Based on a rating when users submit photo comments (give heart=like, no heart=dislike), we classifying images with attitudes and store info in backend to be read when making recommendations for that user. Also, due to the time constraint, there some of these features and concepts that are not fully functioning as implemented in our current prototype. As we prepare for a wider release, these are the immediate items that will be fleshed out.

Accomplishments that we're proud of

Currently, students will explore Columbia Dining's online menu to help them decide which cafeteria to dine at. However, this method lacks democratization and information density than other applications like Yelp have. We are proud to resolve this issue with StuDine which imports the dining hall options and food option for every meal each day and allows students to submit comments about each food option they have at each dining hall, rate different food and dining halls.

What we learned

Materialize Ideas into Solution : We learned about the process of product development from the process of bringing an initial idea of “something that might be helpful” into an actual prototype.

Data Analyzation : Through building an image recognition model, we learned to set up the recommendation system with data analysis.

Design Webpage from Scratch : We explored the functions of Quasar, Vue, and Visual Studio Code as we designed the webpage.

What's next for StuDine

Model Training : In order to provide better recommendations for users based on their photo and comment submissions, we will implement data sets to train the suggestion model.

Calendar Function: We are hoping to also add a calendar function that will allow users to set up time to meet up to meal and decide on dining hall to go to.

Meet New Friends: We plan to add friend function in the next step, so students can share their common love/ hate towards cafeteria and meet some new friends!

Health Reminder: It’s useful to add a health reminder for showing calories information and how much to consume daily, healthy eating tips, and when to exercise.