Common Crawl Foundation

@CommonCrawl

Common Crawl is a non-profit foundation dedicated to the Open Web.

San Francisco, CA

Common Crawl Foundation
@CommonCrawl
Oct 14
Somali was written down on purpose. AI is learning it by accident - Hiiraan Online
From hiiraan.com
Common Crawl Foundation
@CommonCrawl
Oct 14
Findings of the WMT 2026 Shared Task of the Open Language Data Initiative Laurie Burchell Common Crawl Foundation
1
Common Crawl Foundation
@CommonCrawl
Oct 12
The Common Crawl Foundation at the TPDL Conference The Common Crawl Foundation participated in the 30th TPDL (Theory and Practice for Digital Archives) conference in Faro, Portugal presenting two contributions on Crawl theory and on archival data enrichment.
1
Common Crawl Foundation
@CommonCrawl
Oct 12
arxiv.org/abs/2610.13321 Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We investigate how a crawler should allocate its budget across and within hosts to collect more
arxiv.org
Host-Aware Web Crawling for LLM Pretraining Data
Web crawling is a major source of large language model (LLM) pretraining data, yet data-selection pipelines discard a large share of crawled pages. We investigate how a crawler should allocate its...
4
Common Crawl Foundation
@CommonCrawl
Oct 7
Common Crawl - the ultimate time capsule for knowledge | Amazon Web Services
1