How AI systems crawl, rank and cite web content. Each piece is sourced from provider documentation and our own observation, with the sources listed so you can check them yourself.
AI Overviews lean on a specific subset of Schema.org markup to identify entities, products, and answers. Adding the right types is one of the cheapest visibility moves available.
6 sources · Google Search Central
Perplexity operates two crawlers with different obligations under their stated policies. PerplexityBot respects robots.txt; PerplexityUser is treated as a user-initiated browser fetch and is not bound by it.
3 sources · Perplexity Documentation
Google introduced a separate user-agent token, Google-Extended, that controls only AI-product training. Blocking it does not affect Search ranking. Most publishers don't realise that Googlebot ≠ Google-Extended.
2 sources · Google Blog
OpenAI now operates three distinct user agents, each with different purposes and different opt-out implications. Blocking GPTBot doesn't keep you out of ChatGPT search results — and vice versa.
4 sources · OpenAI Platform Docs
