ethicalClothing

little logo
model 1 right before segmentation
segmentation skeleton with nodes
plot of our segmentation model after inserting a picture
model 2 right before segmentation
formula
idea

Inspiration

When we heard the different points of the challenge, we knew that we had to place a lot of effort intrt into coming up with a great use case. Another thing we wanted to achieve was combining sustainability with a use case that favoured keeping the privacy of data. That way we came up with the idea of an app that, given a full body photo of yourself, could identify what you are wearing and score the sustainability of your clothes.

What it does

Our app has two main "sections". First, you open a mobile app (iOS or Android) through which you upload a picture, which is then sent to our server where it will be processed 2 times: First, it will be segmented, plotted over with a silhouette and then split into different images for different parts of the body. Second, our own ai model grabs these images and infers what clothing item are you wearing, to finally return a series of values associated to the item and calculate your final score. The score is calculated with a formula that we've come up with (see in attached images).

How we built it

The system has been built with the following technologies:

The mobile application is built with React Native.
The backend server is build with Flask (Python).
The machine learning tasks have been performed with mediapipe library for segmentation and tensorflow for classification.
To develop and train our models we've used Google Colab's notebooks.

Challenges we ran into

We ran into quite a lot of challenges. First of all, it was the first hackathon for some of our team members, so that's something to keep in mind. Some other problems we have encountered are:

Inexperience in most areas of the project: Pretty much all of the technology used in the project has been new to the person coding it. For one of us it was the first time working with react native, for another it was developing a backend API with Flask, or working with these types of AI models.
Interesting use case... Hard to find models: since our process required 2 models to achieve 2 steps of the task, we tried to find a pre trained model that could help us segment and identify clothing items worn by the model in the picture. That was easier said than done, and we had to change our approach several times until we arrived to our solution.
Complex architecture: Given the nature of the project, we've had to integrate different systems into one, and it was a challenge to juggle all these new technologies and make it work, as well as trying to integrate our models into the Disco platform.

Accomplishments that we're proud of

We are very proud of being capable to overcome the various obstacles we've faced during the hackathon. Having to re-imagine our solution halfway through the hackathon because we weren't able to find or make a model that fit our needs, and coming up in the end with an mvp of sorts is something that we are proud of.

We believe that the more ambitious the use case was, the more complicated it would be to implement. That has proven to be true for us, as we had big goals with this challenge but we're proud to have risen to the task and not having quit when it was not looking good.

Lastly, we are proud of learning all these technologies that were new to us. Maybe taking another approach would have landed us better results, but hackathons are places to learn and we learned a lot this weekend!

What we learned

We have learnt a lot in regards to image processing in ai models, and different technologies altogether:

Developing mobile applications with React Native
Developing a backend that serves an API with Flask
Learning to work with Google Colab's notebooks
Using technologies like NGrok that are perfect for hackathons.

What's next for ethicalClothing

In the future of ethicalClothing, there are a lot of ways to impove the project:

Fine tuning our models to get better precision and accuracy, both for segmentation and score prediction.
Finding or creating better datasets to feed the application.
Improving UI/UX and adding use cases to the application. For example: being able to take pictures directly with your camera.