ColorFit | Devpost

Inspiration

With an interest in computer vision and color theory, we decided to create a project that detects someone's skin tone and suggests some clothing items/colors that would suit their skin tone. We were also inspired by the increasing demand for personal color analysis, most notably in South Korea.

What it does

The user takes a picture of themselves using our web app or uploads an existing image. Their skin tone is detected using computer vision by tracking the color combinations based on Google's Monk Skin Tone Scale, our algorithm chooses matching tops that would suit their skin color. For example, warm colors like red and orange match those who are tanned while cool colors like blue and purple match those who are pale or fair-skinned. Based on that extracted skin tone and the user's body type, our app creates an outfit that could suit the user. Additionally, we have used Cohere & Stable Diffusion to match up other pieces of clothing such as trousers or boots that would suit the chosen top to build together the overall outfit for the user.

How we built it

Web app frontend

Tools used: Next.js, TailwindCSS

Although initially starting with React.js, we switched to Next.js after struggling to set up TailwindCSS with React. Only one of our teammates knew it, but it helped to have this framework to offload the routing, styling, etc.

Skin tone detection

Tools used: OpenCV (including human detection)

Colour Theory

Tools used: Skintone Research @ Google AI

We utilized the Monk Skin Tone Scale, which provides a spectrum of ten skin tone shades that ML researchers use to prevent bias in their models. We researched which color fabrics work well with each shade (making a few assumptions about undertones) and created a small image dataset from which to suggest which top they would look good wearing.

AI building blocks

Tools used: Cohere - Text Generation
Hugging Face + Stable Diffusion - Image Generation

Challenges we ran into

The accuracy of existing models (and the quality of the image datasets) in accomplishing our tasks was not up to our expectations.
Our team split up into frontend and backend evenly, which made coding go faster but filled us with merge conflicts and confusion. We had a lot of trouble putting the two halves together, and getting our Next.js app to talk to our Flask app.

Accomplishments that we're proud of

We created a full-stack web app with AI features from scratch within 36 hours that actually work!

What we learned

Where and how to retrieve datasets and how to integrate them into our codebase
The importance of data cleaning
Doing research with curiosity and initiative

What's next for ColorFit

v3: consider weather or seasonal temperature when recommending an outfit. This can be done through a weather API and some changes to our prompt to OpenLLM. (We imagine this to be seen as an outfit decider to be used as a daily service and to make it more applicable to the real world, integrating the weather is a great way for the user to interact in more detail with their clothing through this medium. )
v4: provide shopping suggestions (e.g. through an Amazon or eBay API) for items similar to what was suggested.
v5: allow users to upload their wardrobes (albeit manually) and suggest outfits only based on what they have.