Jabber - original

Chat with your friends using animated faces instead of crappy video calls.

Comment

Inspiration

After suffering many poor quality video calls, we wondered if we could produce a fun and efficient web-based calling experience without video.

Jabber replaces video with quirky, customizable face animations to accompany voice calls with your friends. No account necessary, just start calling!

What it does

  • JabberStories: Record stories with audio and animated face graphics
  • JabberChat: Live chat with friends with audio and animated face graphics

How we built it

  • We simplified and tracked facial movement and expressions using clmtracker, an open source library developed by Audun Mathias Øygard, based on research by Jason M. Saragih.
  • We connected users' audio streams using simple-peer, an open source library developed by Feross Aboukhadijeh.
  • We used browser-based features to connect to device cameras and microphones.
  • We used Firebase, Socket.io, and Express to manage other real-time features and store users' recorded stories.

We do not store any identifying information about users. Video data is not streamed to other users or stored in the database. Audio data is streamed to other users in the chatroom, but not stored in the database.

The clmtracker model uses a support vector machine on a 20-factor principal component analysis (PCA) to determine the locations of the facial features from camera frames. To display the face graphics, we store 71 points that represent facial features as well as 20 parameters for the model. The PCA processes decomposes the image information to a level where we believe that it cannot be reversed to identify the original user.

Challenges we ran into

  • Computing facial geometry and depth to add custom hats, facial hair, or clothing
  • Wasting time using peer-to-peer libraries for streaming audio that are no longer maintained or functioning

Accomplishments that we are proud of

  • End User Satisfaction: Completed a 15-minute call with a friend in Urbana, Illinois, who described the experience as "honestly really cool" and "extremely dank."
  • Implemented both recorded stories and live chat
  • Restructured the clmtracker library to render face graphics without video stream
  • Created customization options to add clothing to face graphics

What we learned

  • How to stream peer-to-peer audio and reduce feedback
  • How to use blob storage in Firebase to save recorded stories

What's next

  • Measuring if this approach to calling performs better compared to video calls in terms of latency, battery consumption, and sound quality
  • Improving browser support, currently performs best in Chrome and Firefox
  • Add more options for users to customize their face animation
Share this project:

Updates