Jibo: The Understudy

Inspiration

So many people grow up with social anxiety - and levels of anxiety can range from mildly annoying to crippling. We want to provide an experience that will allow a child to grow in their social confidence without pressuring them in ways that make them unduly uncomfortable. There are also people who would like to involve themselves in acting, but for whom that would be impossible. We want to make theater accessible to those people as well because the arts should always be available to everyone.

What it does

Jibo is an intelligent social robot that interacts with the user. In our specific use case, Jibo recites and practices the lines of a play. When Jibo is given a script, he can emote according to a programmed direction. This allows children to practice their lines, but also to be a director. The child and Jibo make a perfect theatrical pair.

How we built it

Jibo was built using the jibo-sdk library. This SDK allows us to build different behaviors for our Jibo. We could then interact with the Jibo in our simulator. In addition to developing the behavior, we also wrote a python script to take plays and translate them into the necessary documents for Jibo to perform the play. We also explored the capabilities of Jibo to handle misspoken lines when the keywords are still present. Using Indico’s Machine Learning API, we’re able to figure out the keywords of a line. Using the keywords, Jibo can ensure the person knows the majority of the lines and users can decide if Jibo should correctly misspoken lines or just proceed. For example, if the line is “Hello, how was last night?” and the user says “Hello, how was your night?”, Jibo would still recognize the line.

Challenges we ran into

While the Jibo SDK has many features, it proved to be much more difficult to work with than we had hoped. There were many bugs and a lot of places where the documentation was out-of-date. We spent much more of our time fighting with the Jibo SDK than hacking. While the Jibo engineers were great support, they were only here for the first few hours and debugging without them proved very difficult. Despite this, we were able to accomplish a hack we are very proud of.

Accomplishments that we're proud of

We are proud to have built an app that can help bring theater to those who wouldn’t necessarily have the option. By giving people a method to build their social confidence, we are attempting to increase quality of life.

What we learned

On our team, we had 5 first-time hackathon hackers and only one veteran hacker. This hackathon was an amazing blend of art and technology, and it was a great space to learn more about hackathons and the cycle of learning they promote. We loved the opportunity to explore a product from all sides of design, business, art and technology. Additionally, we had the opportunity to learn about Jibo and its impressive capabilities. When simply interacting with robot, it is easy to miss all of its impressive capabilities.

What's next for Jibo: The Understudy

Given more time, there is a lot we’d like to add to our project. Currently all of Jibo’s emotions are hardcoded, which works well enough, but we’d love to have Jibo analyze his lines and figure out what emotion to show. We’ve started playing with this using the indico emotion analysis API, but there’s still work to be done. Next, currently Jibo can only respond with the next line. While we have worked with Jibo handling incorrect lines, we haven’t given Jibo the ability to correct the child when they’re wrong.