Cauldron | Devpost

Inspiration

During our plane flight to Helsinki from the United States, each of us had work we wanted to do, but no internet! How were we supposed to be productive with out it?

Some of our coding projects had some blocker that could be solved by programming docs, stack overflow, or even from some random google search result. And others wanted to read blog posts, study for future tests, or just catch up on reddit.

We even tried to hack the airplane wifi out of boredom!

Because of this, we decided to make Cauldron as an AI powered, predictive, offline search engine that will automatically download webpages you are likely to visit in the future for you and save them in the background.

What it does

While you browse, the URL's you visit will be fed into our AI driven algorithms to decide, amongst both linked pages and pages we find from scratch based off of your history, what to cache for offline viewing. Then for each cached page, we generate summaries and indexes. Through our chrome extension, you can search for webpages and open them in your browser. Relative links will work and external links will also work if we have already cached that page, making the browsing experience familiar and seamless.

Features

*Integrated blacklist to avoid sensitive websites like banking

*Auto-detects when Internet is down

*Integrated search for cached websites

*Only downloads pages that load successfully

*Predictive link caching

How we built it

We made a chrome extension and local flask server.

When the chrome extension is turned on, it will send URL's visited to the local flask server.

The flask server will run a multithreaded function that will analyze the URL contents, find similar URLs/linked URLs, rank them using NLP techniques, and save some threshold amount of them with a multithreaded download queue to you device. After download, they are indexed with their generated summaries attached.

From this point on, everything will work without internet:

When you type a search query into the chrome extension, it will query the flask server and our search algorithm will return the most relevant results to your browser to choose from.

Then you can click a link in the search results and access the cached page. It will be loaded from the device into your browser. All internal links will work, and all external links will automatically work if they have been cached.

Challenges we ran into

Iterating on different NLP and search methods to make them as effective as possible. Multithreading functions for performance so that you can cache pages as fast as you visit them. Caching websites is harder than it seems with so many external dependencies.

Accomplishments that we're proud of

Final algorithms, performance, cohesiveness of UX

What we learned

NLP methods, search algorithms, multithreading, wget specifics, browser features

What's next for Cauldron

Connecting multiple users' flask servers so that files can be shared across device. That way all of us could have downloaded different webpages and shared them seamlessly on the plane ride to Hack Junction. This will multiply the value for each individual user of Cauldron.