Common Crawl Foundation (@CommonCrawl) / X

Common Crawl Foundation

1,465 posts

Common Crawl Foundation

@CommonCrawl

Common Crawl is a non-profit foundation dedicated to the Open Web.

San Francisco, CA

Joined February 2010

Common Crawl Foundation
@CommonCrawl
Jul 28
July 2026 Crawl Archive Now Available The crawl archive for July 2026 is now available. The data was crawled between July 7th and July 25th, and contains 2.14 billion web pages (or 364.01 TiB of uncompressed content). We also announce some improvements and changes.
Common Crawl Foundation
@CommonCrawl
Jul 27
Common Crawl Joins Project Tapestry Common Crawl has joined Project Tapestry, a global initiative led by the AI Alliance to advance open, sovereign AI. We will contribute our expertise in responsible web data, multilingual coverage and culturally informed AI development. 📷
Common Crawl Foundation
@CommonCrawl
Jul 21
Measuring Crawled Coverage of a Website in Common Crawl How can we measure how many pages we’ve crawled from a particular website? The answer is a lot more complicated than you might think.
Common Crawl Foundation
@CommonCrawl
Jul 17
Congrats on luca@commoncrawl.org getting 5000 Github stars on his project GROBID! A machine learning software for extracting information from scholarly documents grobid.readthedocs.io/en/latest/
Common Crawl Foundation
@CommonCrawl
Jul 17
Is one vantage point enough? IPv6 across the top million web hosts We probed the top 1,000,000 web hosts for IPv6 from five vantage points on three continents. 327% work from everywhere, but one vantage point turns out to be enough for the headline rate.
See @CommonCrawl's full profile
Sign up
Log in

See @CommonCrawl's full profile