SignKit-Learn

A web app that allows users to practice their sign language skills by conversing with a Microsoft bot!

Inspiration

Best Buddies and social inclusion: One of our team members volunteered with Best Buddies, an organization dedicated to ending the social isolation of the millions of people with intellectual and developmental disabilities. We thought that one way to bridge the communication gap, would be to make it easier for people to practice their sign language skills with a bot!

What it does

SKL allows people to communicate with a Microsoft chat bot using sign language. The web app captures gestures, analyzes them using machine learning, converts them into text, and then sends that text to Microsoft’s chat bot. The chatbot then responds with an appropriate message to keep the conversation going.

How we built it

We built it using the Microsoft Azure services. We used the Custom Vision AI to train 100s images of sign language words. After, we used the Bot Framework to build a chatbot that could take the processed images, and respond! We used Python in the back-end to relay the JSON objects between the different services.

Challenges we ran into

Since no one on our team had ever worked with chatbots before, configuring our Microsoft bot to carry on a conversation was challenging at first. There was also an issue parsing webcam images and communicating with the chat bot at the same time, so we multithreaded the server (which was a good learning experience :D).

What we learned

We learned to work with Microsoft’s custom vision AI, and Luis to train the bot. We learned how to program webcams to automatically snap pictures in Python, integrate JSON objects, and setup a Python server.

Accomplishments that we're proud of

We built our first ever bot using Luis.ai and the Microsoft Bot Framework, our first multi-threaded python server, and managed to dapple in machine learning!