Use augmented reality, facial recognition, and machine learning to create holographic name tags for the people you meet. No more awkwardly forgetting someone's name!
capture.pyis used to build the information and face-matching data sets.- OpenFace is used to process the data.
idknowuserver.pyis used to host the processed data to the network.- Unity / HoloLens App is used to supply streaming video to be inspected and display results to the user.
capture.py is used to build the information and face-matching data sets. It provides text and photo capture tools and automatically puts it into a folder structure based on the user's supplied name. |
 |
|---|
Use the following commands to process the data from capture.py into usable results:
From your shell (it will open an interactive docker container with openFace already configured):
docker run -p 9000:9000 -p 8000:8000 -v C:/idKnowU:/root/openface/idknowu -t -i bamos/openface /bin/bash -l
From inside the openFace container:
| Function | Command |
|---|---|
| Size up the training pictures and extract the faces from the them | for N in {1..8}; do /root/openface/util/align-dlib.py /root/openface/idknowu/training-images align outerEyesAndNose /root/openface/idknowu/aligned-data --size 96 & done |
| Compute the features of the faces | /root/openface/batch-represent/main.lua -outDir /root/openface/idknowu/features-data -data /root/openface/idknowu/aligned-data |
| Train the recognizer | /root/openface/demos/classifier.py train /root/openface/idknowu/features-data |
| Test and see if it works | /root/openface/demos/classifier.py infer /root/openface/idknowu/features-data/classifier.pkl /root/openface/idknowu/test/capture.jpg |
| Start the script that will repeatedly check the filesystem for new face captures | python /root/openface/idknowu/openface_classifier.py infer /root/openface/idknowu/features-data/classifier.pkl /root/openface/idknowu/server/images/latest_capture.jpg |
The last command records and updates the file /root/openface/idknowu/server/ with the face it detects in server/latest_capture.jpg and loops as quickly as possible (waiting half a second on error).
- Run /server/idknowuserver.py on your local machine from the server directory
- Build and install the Unity project to HoloLens or run from the editor
- Note: In Unity use
127.0.0.1as the server if on local device or your actual local IP if from another device - If the file fails to work on first load, you may need to scrap the cache files and rebuild. It saves more than 70MiB to load to GitHub without them, but can occasionally cause some issues on first build.
- Note: In Unity use
This project was created for the Creating Reality Hackathon, hosted 2018-03-12 through 2018-03-14 at the University of Southern California in Los Angeles, California, USA.
- Requirements: Python3, openCV, OpenFace
- idKnowU License: MIT
- Dependencies retain their own respective licenses
https://cmusatyalab.github.io/openface @techreport{amos2016openface, title={OpenFace: A general-purpose face recognition library with mobile applications}, author={Amos, Brandon and Bartosz Ludwiczuk and Satyanarayanan, Mahadev}, year={2016}, institution={CMU-CS-16-118, CMU School of Computer Science}, }
B. Amos, B. Ludwiczuk, M. Satyanarayanan, "Openface: A general-purpose face recognition library with mobile applications," CMU-CS-16-118, CMU School of Computer Science, Tech. Rep., 2016.
