We audited 24,898 of the world's most-visited websites for every barrier that blocks bots: WAF, antibot, CAPTCHA, JavaScript, rate limiting, TLS fingerprinting.
Only 18.5% of the web is truly open.
This is the State of Web Access 2026. What we found 🧵
We pulled the robots.txt files from 24,898 of the web's top sites to see which bots get blocked the most. The entire top six: GPTBot, then CCBot, then ClaudeBot, Google-Extended, Bytespider, and meta-externalagent. Every single one is an AI crawler. Not spam bots, not scanners.
We're talking Agent Security with @Humabound_AI on the 24th this month. Live demos, open conversations, only for those who sign up and join us live: luma.com/wci93kpz
