blossom.me | Devpost

Inspiration

A recent American survey indicated that 64% of Americans wanted to start taking skincare seriously, and in accordance with that, the number of Instagram and Tiktok self-care and beauty tips continues to exponentially grow. However, another survey showed that people have less than ideal face cleansing habits and products. Although there are many products available on the market, personalization and knowledge translation about the ingredients used remains a struggle.

Currently, websites selling skincare products have some degree of personalization, and social media platforms provide information about products for a lay audience. However, there lacks a cohesive application where product information and personalization go hand in hand to provide consumers with a hub of resources regarding ingredients in skin health products.

Even amongst ourselves, our preferences regarding self-care very drastically differs due to several factors. As women with a science and research background, we wanted to serve as a credible liaison between the latest scientific literature and the general public to ensure that all the essential information about your self-care needs are available at your fingertips. This is why we came up with blossom.me, your personal skincare, haircare, and makeup specialist!

What is blossom.me?

blossom.me is your all-in-one self-care and beauty app that provides personalized product recommendations while explaining the science behind the ingredients used. The app allows users to set up a profile for skincare, haircare, and makeup based on their natural features, allowing users to identify their concerns, preferences, and goals through an interactive questionnaire. This includes indicating the desire for vegan/cruelty free products, noting allergens, considering time spent on skincare routines, etc. Once the profile is established, the app uses machine learning and prompt engineering to generate a personalized recommended product list based.

One main feature of the app is a scanner; users can scan a product, or the ingredients list of the product, and in combination with the user’s questionnaire, our app will determine how suitable the product is for the user’s skin type, concerns, and goals.

Selecting products from the recommended tab will lead to an ingredient list that is color-coded based on suitability as well as a short summary about the product’s appropriateness. This would be personalized for the user’s profile and translated for a lay audience. Further clicking on a specific ingredient will lead to a separate page with a short summary about the ingredient that is once again translated for a lay audience alongside scientific articles for further reading.

The app will also have a skincare routine tracker and symptom log form. Users can track the products used in their daily routine as well as any changes to their skin, including acne flare-ups, dry skin, increased redness, etc. This can be used to document the role that skincare products play in changing skin health, and further can be used to determine the best skincare products.

How we built our project

We started with a brainstorming session, trying to identify the problem and issues we want to address through our project. After identifying the gap in knowledge between the lay person and ingredients in skin/hair/makeup products, we discussed what potential software and hardware we could use to solve this problem. As we expanded on our idea for blossom.me, we ran a design sprint. In 2 hours, it moved us from ideation to problem-solving, and it helped us narrow down our target population and specific issues to address. We decided that a mobile application was the best choice for our problem for ease of access to our target audience, and the frequency of use we anticipate for the app.

After this, we delegated tasks in terms of research, design, and coding. Understanding the limited time we had, we leveraged our individual strengths and goals to tackle the project with great vigor. Our implementation during this hackathon focuses just on skin care, so in terms of research, we had to identify what products and chemicals specifically would be good or dangerous for different skin types. Next, we used Figma to prototype our app based on the sketches we came up with during the design sprint. Finally, we implemented the back-end for the “Scan” feature of the app in Python on Google Colab. We used Pytesseract to extract text from images. This would be used to extract the list of ingredients from a picture of a product. Afterwards, we used Cohere to analyze the ingredients list and classify each ingredient based on whether it would be good or bad for a certain skin type. Our implementation only covers the “Dry” skin type, but in the future, this would be done for all different skin types and goals, and also for hair and makeup. Afterwards, our program outputs whether this product would overall be good for their profile.

Challenges we ran into

Coming up with an initial feasible idea that we were passionate about took some brainstorming and collaboration. The sprint took an hour longer than budgeted because we had so many ideas, so discussions often went over our timer beeping. This was also our first time using Figma to design an app, so there was a big learning curve that consumed a lot of our time. We initially wanted to implement the app on Xcode or Android Studio, but we were unable to due to limited storage and RAM on our personal laptops. Instead, we decided to implement a part of the back-end involving the Scan feature, which encouraged us to learn new packages and libraries involving Pytysseract and Cohere. Another challenge is that we did not have any available dataset that classified individual ingredients as good or bad for specific skin types / skin problems. As a result, we used Cohere with examples that we independently researched, and our list of examples was not exhaustive. Going forward, a more exhaustive list of examples for all different skin concerns would be needed for a full implementation of our app. Practicing our pitch together was also a challenge since we had very little time left to create an organized presentation that flowed coherently. Despite these challenges, we’re proud of our end result and the amount that we learned as beginner-level hackers.

What we learned

We are all beginner-level coders, so this project helped us learn a lot of new things. None of us had experience leading product strategy or design thinking sessions either. Having minimal experience with co.here and Figma, we pushed ourselves to learn something new under time pressure. From playing around with the app layout in Figma to changing code in Cohere, it was very nerve-wracking but very fulfilling. It was quite shocking to us how easily we were able to learn to use Cohere and have it tackle our problem of classifying ingredients, even new ones, despite not having a full dataset with all possible ingredients available. In the end, we’re all super excited about what we were able to build.

Although we knew each other before this hackathon, as this was our first time working together on a time-sensitive project, it served as a huge learning experience for us to adapt our working styles quickly to increase the efficiency of the team. We learned how to combine the techniques of playing by our strengths and using this hackathon as a learning opportunity to bring our idea to fruition. We learned to work under time constraints within a team, and how important it was to communicate our ideas effectively to build a cohesive product, and how to pitch our idea convincingly.

Accomplishments

Completing our first hackathon
Presenting an idea that solves a unique real-life problem
Learning and using Figma, Pytesseract, and Cohere for the first time
Having the app work in itself, leading to all the pages and activities that we wanted to include

What's next for blossom.me

Invite specialists such as dermatologists or dermatology researchers to access user progress and verify app recommendations
Add a consultation page where consumers can reach out to other community members for advice or seek professional consultation from specialists
Within the routine feature, implement a facial photo logging system that uses AI/machine learning technology to note blemishes and day-to-day changes in skin to provide more personalized advice
Expand to haircare and makeup as we were not able to present these as examples
Incorporate financial personalization features such as price-matching and filters on the recommendations page
Integration with Apple Watch, Health App