dragonfl.ai

An innovative machine learning model that empowers the visually impaired to perceive their surroundings using auditory cues.

Comment

Inspiration

During our research, we discovered that 5.4% of Canadians aged 15 and older report having a visual disability. This sobering statistic underscores how many people face challenges due to visual impairments. Inspired by this, we aimed to integrate AI and text-to-speech technology to enhance accessibility. We brainstormed ways to empower visually impaired individuals by allowing them to "see" through hearing.

What it does

The dragonfl.ai headset is made up of three major features: facial recognition, object identification, and voice commands. The features are tied together with text-to-speech to fully bring sight through sound.

How we built it

We built the software for dragonfl.ai using Python, OpenCV, Google Cloud Platform and a handful of other tools such as Numpy, Pickle, etc. We also built a web dashboard prototype that displays analytics of the user’s activity and interactions. All can be found in the source code.

Challenges we ran into

The main challenge during the production of dragonfl.ai was integrating the Google Cloud API and functions with a full-stack app created using Flask. To create the login, we used SQLAlchemy, an object-relational mapper of SQLite3. However, to store the statistics of each user, we used Firebase. Firebase’s SDK allowed us to seamlessly integrate user data into Google Cloud, through a user-friendly SDK. We chose Firebase’s SDK as a middleman to Google Cloud rather than the GCP SDK because it was more user-friendly and ideal for web applications (we used Flask). The use of various databases made it more difficult to manage user information. We also had several issues with facial recognition with its mismatching users. By adjusting the threshold through trial and error and being more efficient in memory usage, we were able to integrate a smooth facial recognition software ALL RAN on CPU threads.

Accomplishments that we're proud of

We’re proud of our ambition and passion for solving a global problem. It was remarkable how we collaborated and merged several features to create a functional and practical product. In addition to the accessibility and physical project, we are also proud of the technical aspects and the challenges we’ve endured. While we had experience building projects with GCP and Firebase, this was the first time we integrated the service with OpenCV (the past was web dev projects).

What we learned

During this hackathon, our team delved into various libraries and frameworks, such as Flask and Pickle, to enhance our coding capabilities. These tools enabled us to streamline web development and data handling processes. We also learned about many new APIs, like leveraging Google Cloud and OpenAI for swift and powerful AI applications. Google Cloud offered versatile cloud-based resources, storage, security, flexibility and reliance while OpenAI's API empowered us with advanced natural language processing capabilities that were able to understand the labels, interpret the context and announce them. This hackathon experience has fueled our passion for learning and innovation in the dynamic world of programming and technology.

What's next for dragonfl.ai

We want to continue testing the capabilities of dragonfl.ai past this demo. After recognizing the potential to break new ground and deliver solutions to visual impairments, we feel intrigued and driven to continue building it out.

Hardware

In the future, we plan to create a headset that will hold the software components as well as the camera to detect the surroundings. Starting, we would create a large headset that would faintly resemble a VR headset. As the company progresses, we plan to reduce the weight and bulkiness of the design to create sleek glasses and reduce potential issues such as minor motion sickness, etc.

Hosting SQLite3 Login on Google Cloud

At the moment, the Login system created with the Flask login library is not hosted on Google Cloud. This is mainly because of the high costs of deploying a SQLite3 server. Moreover, we must figure out the logistics regarding multiple databases and interconnect them together.

Streamlit Integration

Provide graphs that update in real-time based on the Firebase Database.

Built With

+ 1 more
Share this project:

Updates