Bookmarks.ai | Devpost

Inspiration

It all started on Galentine's Day when our team found ourselves in a frenzy trying to track down a cake recipe we had seen on Instagram. Despite our best efforts, we couldn't locate the post amidst the sea of saved content across platforms like TikTok, Instagram, and Twitter. This experience led us to a realization: the lack of a robust search mechanism for managing our social media bookmarks was a widespread problem.

Bookmarks.ai revolutionizes the way you organize and search through your saved bookmarks across social media platforms.

What it does

Bookmarks.ai serves as a platform to aggregate your saved posts (bookmarks) throughout your social media platforms and then runs a semantic search on the captions and hashtags of the post to retrieve relevant posts to your query when you are searching for specific content. In addition, it leverages k-means clustering and sentence similarity to group similar posts together in clusters, which are available on the home page for easy access and browsing.

How we built it

We began by customizing the Instagram API which collects post metadata such as the image link, the web link, the caption, and the post ID. From there, we stored this information in the format of a JSON file.
These captions were then split using a CharacterTextSplitter, embedded with Langchain, and the vector embeddings were stored using MongoDB Atlas.
After creating a vector search index in MongoDB Atlas, we were able to successfully run queries with semantic search and the post details of the relevant posts would be returned as the result.
Based on the array of post captions, we were able to run K-Means Clustering using sentence similarity based on a model we found on HuggingFace to divide the posts into clusters. We were able to leverage prompt engineering and an OpenAI LLM to automatically come up with names/descriptions of each cluster (such as "cooking" or "fashion").
Our front end was built using React.js, HTML, and CSS

Technologies and Frameworks: React.js, HTML, CSS, Python, LangChain, HuggingFace, MongoDB, OpenAI

Challenges we ran into

One of the biggest challenges we ran into was Instagram authentication and being able to scrape Instagram-saved posts. Since most social media apps have strict authentication guidelines, accessing embedding URLs for each Instagram post was often difficult. There was also a rate limit to Instagram's API. Additionally, all of our teammates were new to semantic search and k-means clustering so it took us a while to figure out how to create vector embeddings and the vector search index. It was also difficult to integrate the backend with the frontend as our backend would often return JSON files consisting of web links to posts, and we faced difficulty converting them to embedding codes as the Meta Developer Tools API that was used to do that was often unpredictable in generating access tokens.

Accomplishments that we're proud of

This was our first time exploring semantic search and k-means clustering. We were excited that we were able to build a project with both of these features fully working by the end of the project. We were also excited about the problem we were solving—we had personally faced challenges managing our bookmarks and it was cool to use LLMs in a meaningful way to make that process significantly easier.

What we learned

This was a great opportunity for all of us to learn about new frameworks and technologies we hadn't used before, become comfortable debugging them, and quickly come up with a backup plan when things weren't going smoothly in the end. Ultimately, we were able to create a product that we believe would be personally useful to us.

What's next for Bookmarks.ai

We would love to include AI agents that allow you to automatically create action items from posts that you save and query (think creating a grocery list if you view a baking post). We would also love to be able to intelligently process the images in posts and search based on that as sometimes captions and hashtags may not be enough information.