Rebust! | Devpost

Inspiration

We were inspired by the modern AI's we've learned about in class, and in particular the advanced AI's of today that are incredibly effective at recognizing photos. When we glazed over these topics, we learned that there has not yet been any program developed to understand rebus puzzles, due to the many different and often over-lapping clues that need to be distinguished. Together, it was our goal to get the ball rolling in this area and try to use existing API's to make a web app capable of these functions.

What it does

Rebust! Takes user input in the form of image files, text, and urls to images and combines them in a series of words that is most fitting to the situation, spitting out a probable solution with several backups just in case. The way the back-end works, is that we have several inter-connected python scripts that work off of the Clarifai API, using parsing the images and then using generated recognition word lists to check the validity of the generated word lists and then sort them in order of most to least likely word. The word is then returned to the user.

How we built it

We started with Pycharm, working through the different possible API's and then splitting up the work, having one person dedicated to front end, one to back end, and one going in between helping marry the two. We wrote our web app using Flask as our framework, as well as API's including Clarifai, datamuse, and NLTK. Clarifai is an image classifier API which we used for the image components of the puzzles, and datamuse and NLTK are language API's we used for classifying the phonetic words within the puzzle.

Challenges we ran into

Originally our team used Tensile as the image recognition API which made us run into several road blocks, since it would not recognize general idea, but rather more specific descriptions. In addition, the upload functionality gave us some trouble, because it was our first time using Flask and Pycharm to develop web-apps so it was a learning experience, taking out more established knowledge in Python and applying it to this new IDE and a totally different working experience.

Accomplishments that we're proud of

It works! Our rebus puzzle decryption is able to recognize compound words and return results with relatively high confidence. Considering that over six decades of AI research has thus far been unable to solve this problem, we are incredibly proud of our accomplishment of at least getting partial functionality in this project.

What we learned

We learned how to use the nltk wordnet kit and datamuse for speech and voice recognition as well as making the less concrete topic of neural connections and baseline recognition translate into definitive words and phrases. We also were able to get a more in depth experience with ML using Clarifai and some of our own ingenuity.

What's next for Rebust!

Next we really hope to first and foremost edit the working code and ensure that it is functional for the more complex rebus puzzles and then fix up the UI a bit to make it more fancy looking. We would also like to be able to involve more ML in our puzzle solving in the future so that it can further improve its results with more use.