Selfie Monsta | Devpost

Facial recognition and score
Snapping a photo
Recording mode!

Inspiration

Embarrassed of your student ID card? Do people ask you who’s on your ID card? Face it, most of us have had trouble taking selfies in our lives, and all we can do is suck it up. We wanted to help the many people in the world that face this problem everyday by providing an app that can help you find your best angle.

What it does

Selfie Monsta provides a live time score of how good your current camera capture is to help you find your best angle and look. In order to start using the app, just point your camera to a human face, and once the face has been recognition, the app will run your images into a machine learning model and tell you how good your selfies are.

The app supports two modes of capturing selfies, Single mode - wait until you see a selfie score you like, and click the snap button. You’ll be given the options to redo a photo or download what you have. Smart mode - hold down the snap button, and Selfie Monsta will keep track of your best selfie over a period of time. A small screen will pop up in the corner, showing your current best selfie, and when you’re satisfied, you’ll be given the option to take more photos or download your dashing profile.

How we built it

The front end of Selfie Monsta is built on HTML5, CSS, and JS for fast prototyping of a proof-of-concept user interface that is cross platform. A computer vision library (Tracking.js) was used for efficient face-detection and pre-processing of raw pixel data. The pixel data from the video was then sent over to a AWS EC2 x8 GPU Instance for the prediction of the selfie quality.

Behind the scenes, the image is transformed and fed into a neural network — a modified MobileNet Convolutional Neural Network that does binary classification, which rates the selfie-quality of the image, and outputs a rating from 0 to 1. This is powered by a Tensorflow backend, trained with the Selfie Dataset from the UCF Center for Computer Vision, a labelled dataset of over 40,000 images and respective popularities.

Challenges we ran into

CSS Problems: ..* Dynamic positioning and formatting of UI elements ..* HTML5 elements like canvas and video Making the app look nice
Camera and computer vision APIs: ..* Different devices had different video qualities and resolutions, hard to achieve consistency and smooth performance
Networking: ..* Security issues (Cross Origin Requests, HTTPS), relatively large size of payloads, and high frequency of requests
Transforming images to be compatible with the training set
Training a satisfactory model: ..* Noise and outliers in training set

Accomplishments that we're proud of

We were surprised but also really happy that our machine learning model came out to work reasonably well, without having too many complicated technical components or costing too much of our time by requiring too much data or learning time. We were also able to make the web app be compatible with mobile even though it was originally just for desktop, which was a pleasant surprise since we thought about making a mobile app in the first place (but didn’t feel like we had the skills to do so in a hackathon). Finally, we were very impressed by how our entire project just fit together, from the machine learning model creation to hosting the model calculations on a server to the actual web app connecting packages, requests and the UI together to create the final usable app for the user.

What we learned

Cross-platform development is tricky to get right and meet expectations. Using the right front-end layouts and debugging tools makes a big difference in achieving consistency in design, functionality and performance.

On the machine learning side, we discovered that models can be very volatile, from overfitting to false positives and a host of wrong learned assumptions. We had to learn special techniques to account for these errors, to create a model that does as expected but is robust at the same time.

What's next for Selfie Monsta

We hope to make a native mobile app instead of a web one so we can support more features, and have a smoother UI overall. We also hope to improve the current model with more training and features, or explore other architectures for the neural network. Finally, we hope to be able to apply the concept of Selfie Monsta in the automation and streamlining of processes of substantial photography tasks, such as photo-ID creation or satellite imaging.