Caption

Wearable Translation from Speech to Text!

Comment

Inspiration

The inspiration for our project was the accessibility that it would provide for individuals. Our Caption glasses are for individuals who are hard of hearing or for people who just like reading subtitles. This project is to help people listen better as many people in the world struggle with hearing and we hope to provide a simple solution for that. With these glasses, we hope to bridge the language barrier gap between people in order to facilitate communication. Ideally, we hope that people will use these glasses when they need to contact emergency services that speak predominantly English in order to have the best service to take care of themselves.

What it does

How we built it

We first started off with a schematic which included an ESP 32 Cam, a MAX 9814 microphone and amplifier, an OLED Display, and an FTDI programmer. We were able to use the programmer to run code that would display text which would serve as our “subtitles”. We also utilized CAD to 3D print two different casings, one for our breadboard and the other as a holder for our mirror, reflector, and the OLED display. The OLED displays the subtitles from the audio received by the microphone display.

Challenges we ran into

Justin - There were many challenges that we ran into along the way. Our original plan was to use a concave lens to magnify an image (subtitles from the OLED), but we had trouble calculating the correct image and object distances since our knowledge of optics was limited. As a whole group, we ran into issues where our microphone was not working properly as intended due to either the WiFi or the technical capabilities of the chip in the microphone.

Emily - One of the biggest challenge that I encountered, being on the hardware team, would be having to get rid of the glasses at the end of the project. For the first half of the hackathon, I put a lot of time and effort towards figuring out the holder for the lens, mirror, and OLED screen to get the best display. Not only did I have to figure out the optics lengths and focus points, but I had to have a lot of trial and error since the calculations were not always reliable, especially since I was trying to adjust it to fit onto the side of the glasses. When I figured out the optics, the cadding part had an issue with the dimension—I didn't factor in the thickness of the leg of the glasses, so the reflector was completely off my focal point compared to when I was testing it without the glasses. As a result, I had to make a decision to get rid of the glasses, because after thinking for a while, I realized that the highlight of the hardware wasn't the glasses (even though it's advertised in the name), but rather the fact that you can see captions while looking straight ahead. The glasses frames themselves weren't as important as the lens, which allows you to do that.

Dylan - I helped deal with the majority of the hardware issues, serving as the one to check over schematics and called in to deal with difficulties found during the hardware portions. The two single most devastating challenges I encountered were both from the limitations of the microcontrollers we selected for the task. For one, the ESP32 cam is unable to connect to WiFi and read data from our microphone. As a result, I made the decision for the hardware team to modify the circuitry schematic on the fly as we switched to using a UART communication protocol between two different ESP units—one devoted to WiFi and the other obtaining data from the microphones. However, this was difficult to test and implement properly and although we succeeded in not frying two perfectly good boards, we could not fix this issue. The other issue was that the ESP32 would connect to WiFi but would not talk to our deployed backend. We called over several mentors and I think the hesitant consensus was that the ESP32 HTTP request client could not use Host headers which caused our deployed backend to reject all our attempts to make requests to it. Lastly, we ran into an issue with the recording of audio via the MAX 9814 microphone. Due to the ESP32 Cam hardware limitations, we could not increase the sampling speed (aka the analog read speed) so our audio captured ending up being mostly garbled, barely discernible. Overall, many of the difficulties encountered were well beyond our control.

Accomplishments that we're proud of

Justin - I am proud of how we worked as a team. Although there were some issues along the way, we were able to resolve most of them. On the hardware side, I was proud of how our schematic worked as intended and how we were able to work together with the software members to display text on our OLED screen.

Casey - I’m proud of how we were able to push through many of the hurdles that we faced. I would say that our key word for this project is “pivot”, as many of our ideas had to be revamped or scrapped as a whole. Even though we faced many difficulties throughout this Hackathon, we learned a lot in both the hardware and software team as we were able to tackle concepts that we wouldn’t normally learn in class.

Dylan - I’m most proud of how we were able to adapt on the fly. When insurmountable issues arose, we kept going at it and made steady progress. We ended up getting much further than expected for a team of people learning a lot of new things at once! I’m also proud of how much more comfortable I got with designing and modifying circuit schematics from scratch, especially as a Computer Science major who hasn’t taken many circuits courses at all.

What we learned

Justin - As an electrical engineering major, I never envisioned myself at a hackathon because I did not think that my skills would be applicable. I learned that as a team, we could create a hardware project that would incorporate my knowledge of circuits and integrate it with my teammates who were more familiar with the coding portion. This was my first time developing a schematic from scratch and researching what components (microcontrollers, displays, microphones, wires) that we would potentially need.

Emily - One of the biggest things I've learned during this hackathon is what learning to adapt in projects looks like. For our project, we had portion/dependence on hardware --> the issue with that in which we didn't predict/plan too well prior, would be the amount of trial and adjustments we'd have to go through because of that...because unlike a largely software based project, the materials we had, what we could use, when we could 3d print, etc were all limited and huge factors we had to take into consideration when it came to planning out our time. Our time was also dependent on the software and vice versa, so it was time consuming when one party came into an issue, which isn't extremely ideal for a hackathon with a 36 hour time limit. If one team had issues, the other team would have trouble at times moving forward as well. We only had one shot at 3d printing our parts and we were particularly rushed in cadding the parts since optics took a while to figure out. So not only did we have to adjust the project a lot in response to errors (such as issues with dimensions of the parts), but we also had to think deeper into the root of the project itself, to figure out what we needed to save and figure out (parts) and what we could throw out/put away as it was less important. In fact, I would say that one of the biggest lessons I learned overall was to have a clear goal on what we wanted for the final project --> because not everything will go right during projects, and especially during a hackathon.

Dylan - I learned a lot more about creating a circuit schematic from scratch and using the ESP32 libraries to connect to WiFi and interfacing with modules. I also, most importantly, learned that not all microcontrollers are made the same and that extensive research before selecting one for a project is immensely important. I needed to research my options just as carefully as I research future frameworks and libraries I want to pick up.

What's next for Caption

From a hardware standpoint the next thing we could do if we had more time would be to build on our current prototype. Instead of breadboarding our circuit, we could solder our circuit which will result in more reliable connections and not take up as much space. We could also add a lens to our prototype to magnify the subtitles based off of more research into optics. Finally, we would replace some of our components because women had limited functionality such as our microphone which had trouble picking up sound and our ESP 32 cams which could not connect to the WiFi and produce analog outputs at the same time.

Built With

Share this project:

Updates