According to the World Federation of the Deaf, there are around 70 million people worldwide whose first language is sign language. Gallaudet University estimates anywhere from 500 000 to two million ASL speakers in the US alone. This is a large group of people.

Seeing the lack of support for translation between oral and signed languages, Clive Chan, Colin Daly, Wilson Wu, and I decided to do something about it. We spent a weekend at McHacks, a 24-hour hackathon held at McGill University in Montreal, Québec, building a project we felt would aid in sign language communication.

Logistics

Having brainstormed in advance, we began gathering supplies for our project. Wilson and Colin bought wood, servos, and string. Colin also 3D-printed hand “bones” so that we would be able to get the flexibility of a real hand. Unfortunately for us, no one told Colin that people usually have one left hand and one right hand so he accidentally printed two left hands. Luckily, he caught his mistake a couple of hours before we had to leave and quickly printed a fragile right hand (nothing hot glue couldn’t fix).

When we arrived on the day of the event, we lugged all of our supplies into the building and were told we had to head to opening ceremonies. We figured that this would be a quick walk to a neighbouring building, but we were wrong. The ceremonies were held in a building almost all the way up a very steep hill; this would be an obstacle in the future, but we didn’t know about it yet.

👌… ✌… ️👆… Hack!

When hacking began, we wasted no time. Clive and I created a couple of GitHub repos and began coding. Clive got to work playing with the Leap Motion API: detecting movements and writing them into a format that could be translated back into hand movements through servos. At the same time, I set up Node (incorrectly) and began working on the code to control the servos.

Meanwhile, Wilson and Colin decided to get to work on the hardware side of things. They began by assembling the box that would encase our hands. It was at this point when Colin realised that he didn’t bring a hammer with him. After much panic, Colin and Wilson decided to use a cast iron clamp as a makeshift hammer for the weekend.

As they hammered nails into the wood, Clive and I hammered our fingers into the keyboard. Every once in a while, Clive would wave his hands over the Leap Motion like a wizard or the servos plugged into my breadboard would rotate. Things were going according to plan.

As is often the case in hackathons, the smooth sailing didn’t last for long. Something was missing. Wilson and Colin had finished the box and began putting tendons (nylon string) into the 3D-printed bones for the hands. As they put together the right hand, they noticed there was a missing distal phalanx: an important part of the index finger. In true hacking fashion, Colin pulled out a piece of wooden dowel and an X-Acto knife and began to whittle a distal phalanx.

Eventually, both hands were created and some code was written. It was time to put the hardware and software together and see if we could make the hands act like hands. Of course, going with the theme of the weekend, something was missing for us to test the hands. Colin had forgotten the power supply cord at home which meant that we couldn’t test until we could find a desktop power cord.

Colin and Wilson went around to various buildings on campus in search of the cord that would allow us to test the hands. They ventured near and far, but had no luck. We called all of the local computer repair stores but all of them were either closed or didn’t have what we were looking for. Eventually, Colin and Wilson went to the first floor and spotted a power cable tucked behind the front desk of the building we were in. An organiser gave us permission to take the cable and we could finally test our hands!

Next we had to get voice transcription and x/y/z-movement for the hands working.

Since Nuance was sponsoring this hackathon, we decided to use their API for the voice transcription. Clive went over to the Nuance table and spent most of the night there, figuring out how to recognise the spoken English.

Meanwhile, Colin and Wilson began putting the arms into the wooden box they built earlier. I took a quick nap and when I woke up Colin and Wilson had put the arms in place and began wiring up the servos. I joined in on the wiring action. Colin and Wilson then went to sleep while I finished hooking all of the wires up to the breadboard.

When Wilson and Colin woke up, we got to work on calibrating the system. Unfortunately, we were using an Arduino Mega which, despite having 14 PWM ports, can only handle 12 servos at a time meaning that we would have to use two boards to control all 14 of our servos.

We rewired everything so that we were using two Arduino Unos (which we discovered after having rewired things could only handle 6 servos each — less than the 7 each that we needed). Since we now had to control two boards, my code had to be changed. Unfortunately, the Johnny Five library isn’t extensively documented for multiple boards (or perhaps we were having trouble comprehending things on little sleep). There was an issue with the library accepting an undefined value and not throwing an error, causing us only to be able to control one board — after a stressful hour or so, we were able to resolve it and control the servos again.

At this point, time was running low. After all, the hackathon was only 24-hours in length. Now all we had to do was get the voice-to-sign-language translation working and the project would be complete.

Clive returned from the Nuance table and began testing his voice recognition software and his sign language mappings on the hands. Colin and Wilson also began coding words in sign language for our robot to say. With just minutes before the end of the hackathon, our project signed “hello”. We had done it!

During demos, we got to talk to a lot of people about our project and it was cool to hear the thoughts of the other hackers. One hacker (I didn’t get a name) mentioned how where he was from there was a large lack of funding among the deaf community as far as teachers and translators and that something like our project could help alleviate this. Others thought it was cool and some just asked us to make it give the middle finger.

When judging was over, we were notified that we had made it to the Top 3 and that we were being asked to present at the closing ceremonies. We were pumped but then realised it meant we had to lug our machine up the giant hill to the closing ceremonies.

Being careful not to dislodge the wires, Wilson and I grabbed the box while Clive took the power supply. Colin carried his laptop. We slowly hiked up the hill to the closing ceremonies and, after what felt like hours, we arrived at the closing ceremonies. We presented our project and made some hand puns. We even came away with first place!

So, what did we learn?

McHacks taught us a lot of things. The most important of which was: where there’s a will, there’s a way. So many things went wrong on the weekend, and it would’ve been so easy to give up, but, we pushed through it all and came away with a really cool project!

Some other things:

• We learned some basic words in sign language
• NodeJS is good for absolutely everything
• A distal phalanx is a bone in your hand
• I’m pretty good at trigonometry

We definitely want to keep improving this project in the future! If you like this project, feel free to check it out on Github and Devpost or share this with your friends!

A couple of friends and I discovered this issue and decided we should jump on this project before another big company discovered this amazing opportunity. As a result, we spent a weekend at ECHacks, a 36 hour hackathon hosted by Trent University, bringing our concept into the real-world. Here’s the story of how we created the Sandwich-o-matic.

ECHacks 2016: A Sandwich Odyssey

We arrived bright and early to the event with our trusty toaster, some PVC pipes, a pane of sheet metal, cheap wood, and a dream.

When hacking began, we realized how ill-prepared we really were. Sure, we knew that we were going to use the PVC pipes to dispense toppings, but how could we drop one slice of meat instead of the entire sausage? In fact, how could we even dispense toppings at all? Or get the bread from the toaster to on top of the toppings? Or link the dispenser to our ambitious voice-controlled ordering website? We had to start from scratch.

We began the project by assembling the PVC pipes into a ring. Then, after much brainstorming, we decided we would split each pipe into four chambers and implement a rotating slot on the bottom of each pipe to dispense its contents one chamber at a time.

Once we had the ingredient dispenser figured out, Wilson, our sheet metal expert, got to work on the dividers. This involved ridiculous amounts of hammering to flatten the metal. We figured that hammering by our table would annoy too many people in our room, so we sent him out into the hallway. As a result, he managed to annoy pretty much every single hacker and sponsor in the building simultaneously.

As Wilson was hammering away in the hallway, Colin and I began figuring out how we would control the rotation of the ingredient dispenser. The entire dispenser weighed around ten pounds and was mounted on a single servo motor, so accurately rotating massive pipes was a challenging task to say the least. I hooked up a Photon and Arduino to the servo motor and after lots of tedious calibration, we were able to rotate the barrel to the different dispensing positions.

As the hacking wore on, we fixed old problems, created new ones, wrote code, and grew tired. Wilson was still hammering away in the hallway, Clive was working on the backend, and Colin and I were tinkering with electronics throughout the night. When faced with challenges, we worked even harder.

Finally at 6:57 am, 34 hours after we started, we did the impossible. Against all odds, we made our very first sandwich.

We continued to refine our Sandwich robot until we came up with the final version. Here are two demos of it working:

Demo 1 features our NFC menu. Simply tap your phone to the menu item you wish to order and Sandwich-o-matic will do the rest.

Demo 2 shows our voice-control features at work. Simply say “make me a sandwich with sausage and peppers” and Sandwich-o-matic will fulfill your dreams. (Shoutout to Clive for making all of this work as well as it did!!)

Overall, we’re ecstatic with what we produced. It went above and beyond our expectations and definitely captivated many hackers, mentors, judges, and visitors. In the end, that’s the goal of hackathons: getting more people interested in technology.

Our takeaways

Coming into ECHacks, we wanted to automate sandwich-making. We believe we succeeded in that. Here are some other things that we learned:

• Every small thing can be a big issue

There were a ton of little things that we had to consider when making the machine. For example, getting toast to drop on top of the toppings was annoying. We needed a system that could elevate the top bread until we needed it, but it couldn’t be in the way of the condiment dispenser. It also had to be able to release the toast on command. It took a good amount of time to actually come up with the idea for the two-level slide and gate, something that appears to be pretty straightforward.

• Hardware is hard, but rewarding

This was probably the hardest hack I’ve ever made at a hackathon. We spent hours brainstorming solutions and then even longer implementing and troubleshooting them. Even at the final demo, failure was a possibility, since the machine depended on the cohesion of many independent components.
At the end of the day, however, we made something that looks awesome! It was so rewarding to stand there and watch the machine make sandwiches on its own, and we came away excited about what we had pulled off.

Other things we learned:

• Cutting sheet metal is very time consuming
• Hammering sheet metal at 3 am will get lots of cold stares
• APIs can be really hard to work with (looking at you, Google)
• Peterborough has really good poutine

I would just like to extend a huge thanks to all of the organizers, volunteers, and sponsors who made ECHacks so much fun. Without you, our dreams of taking over the automated sandwich-making industry wouldn’t have been possible!

If you’d like to see more of our project, check out our GitHub repo or Devpost!

Reinventing the Sandwich was originally published in Major League Hacking on Medium, where people are continuing the conversation by highlighting and responding to this story.