EYESSISTANT | Devpost

Inspiration

Although we did not have many ideas at the beggining, we found interesting the challenge of restb.ai as it could diverge in many different projects. One of our colleagues always had the idea about an application/program that could help people with blindness with their surroundings, and we like how this idea could be implemented with the API that restb.ai was offering us.

What it does

The program collects the data from the mobile phone of the person that is taking a video in real life. From there it obtain frames and the Restb.ai API can detect in those frames some key characteristics as the room type or the objects that are in there. Using the Alexa app, the person can ask to know in which room is it, which is the condition of the room or which objects are there.

How we built it

At first, we learn about how to use the restb.ai API, and with that, we extracted some data about different pictures. With this data extracted from de Artificial vision API and also a pre-trained Deep Learning model called YOLOV4 Darknet, we got the data about which room is currently filming, how good is this room and also different objects around the room. The data extracted from this topics, were converted into a .json file that was sended to Alexa throw a public host. Using Alexa developer console, we created a Skill in which we invocate intents with utterances. This let us implement the answers that Alexa can give to the questions of the users. We got the data bridge from our Python script to Alexa throw ngrok libraries.

Challenges we ran into

We found ourselves many times in problems that got us struggling. The first challenge that we faced was to connect the phone camera to our Python software and get frames every 5 seconds. It was not an easy task because we didn't have a normal router WiFi to connect to and also we made it with an IOS mobile. Also, at first we used URL pictures but we made throw base64 images instead of URL and It was not an easy task to implement in our software.

Furthermore, we found difficult to understand at the beggining how to work with the developer console of Alexa as it was our first time doing something like that and we have difficulties to "connect" Alexa to the code where the API was found as we were not sure how proceed. However, the biggest challenge that we found and we could not solve was the class ObjectBoolIntentHandler from the lambda_function of Alexa as we were trying to return if the object that is ask is in the room but we do not get any response from the code.

Accomplishments that we're proud of

We are proud of how far we have got as we have acquire new knowledge working with new programs and new problems that we did not found before. This topic was really interesting because It's a great way to automatize things with Alexa and software web app's while also creating software that could help disabled people. Also, we are really happy because our app worked as expected (after fighting for hours).

What we learned

We learned how to connect an IP Camera from IOS to Python software and how to use the AI API. Also, we learned to implement the YOLOV4 that currently is the best pre trained Deep Learning model. Additionally, how to data bridge from our software code to the Alexa Skills and how to create these Alexa Skills.

What's next for EYESSISTANT

We would like to implement the class that was not working (ObjectBoolIntentHandler). Moreover, it would be interesting to expand this software in environments outside the houses.