Common Crawl Foundation

@CommonCrawl

Common Crawl is a non-profit foundation dedicated to the Open Web.

San Francisco, CA

Common Crawl Foundation
@CommonCrawl
Sep 2
Common Crawl Rate Limit The CDX index server is heavily rate limited. What triggers a block, how long it lasts, and the interface that has no limit.
2
Common Crawl Foundation
@CommonCrawl
Sep 1
A Content Analysis of llms.txt Files from the July 2026 Crawl Archive We analyzed the contents of 584,107 llms.txt files from the July 2026 crawl. Two thirds are produced by a plugin, half follow the structure the specification defines, and a few files even contain prompt
1
Common Crawl Foundation
@CommonCrawl
Aug 29
Thom Vaughan of Common Crawl, speaking from the Paris Open Source AI Summit recently. opensourceaisummit.eu
00:00
Common Crawl Foundation
@CommonCrawl
Aug 26
Common Crawl gets a lot of opt-out requests. We respect publisher rights, even if they don't possess time machines go fix their robots.txt files from 20 years ago. It is what it is. We endeavor to be polite, however. commoncrawl.org/blog/common-cr… Our ongoing opt-out registery /
Common Crawl - Blog - Common Crawl Foundation Opt-Out Registry
From commoncrawl.org
1
Common Crawl Foundation
@CommonCrawl
Aug 25
August 2026 Crawl Archive Now Available We are pleased to announce that the crawl archive for August 2026 is now available, containing 2.14 billion web pages or 360 TiB of uncompressed content.
3