Indexation Internet par robot

Jetpack/12.1; WordPress/6.2; http://site40299152.com

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/153.0.8010.52 Safari/537.36

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.52 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 ForestEngine/1.0 (+https://forestengine.net/#opt-out)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.47 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Reflectionbot/1.0; +https://reflection.ai/bot) Chrome/151.0.0.0 Safari/537.36

Mozilla/5.0 (compatible; PublicWWWBot/1.0; +https://publicwww.com/bot.html)

Mozilla/5.0 (iPhone; CPU iPhone OS 7_1_2 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Version/7.0 Mobile Safari/537.36 (compatible; Bytespider; https://bytedance.sg.larkoffice.com/docx/K5bxdypulop3IIxrJb0lOjLVgFe)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; HubSpot Crawler; +https://www.hubspot.com) Chrome/131.0.0.0 Safari/537.36

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 (compatible; HubSpot DataHub; +https://www.hubspot.com)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/153.0.8010.36 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Mozilla/5.0 (compatible; webatlabot/1.0; +https://webatla.com/bot; abuse@webatla.com)

Jetpack/12.0; WordPress/6.3; http://site13032253.com

panscient.com

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.3122.188 Safari/537.36; compatible; Google-Extended/1.0; +http://www.google.com/bot.html

Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.333.67 Safari/537.36; compatible; Bytespider; +https://zhanzhang.toutiao.com/

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1 Safari/605.1.15; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) Chrome/128.0.7734.125 Safari/537.36

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:121.0) Gecko/20100101 Firefox/121.0; compatible; Applebot/0.1; +http://www.apple.com/go/applebot

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.5403.112 Mobile Safari/537.36; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) Chrome/151.0.4160.117 Safari/537.36 Edg/151.0.4160.117

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/147.0.2579.34 Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) Chrome/124.0.7598.143 Mobile Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.5202.125 Mobile Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:78.10) Gecko/20100101 Firefox/78.10; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.4198.147 Mobile Safari/537.36; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.8674.164 Mobile Safari/537.36; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Chrome/126.0.2670.30 Safari/537.36 Edg/126.0.2670.30

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.3 Safari/605.1.15; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot) Chrome/128.0.663.227 Safari/537.36 Edg/128.0.663.227

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.3176.225 Safari/537.36; compatible; ChatGPT-User/1.0; +https://openai.com/bot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.6899.62 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.4992.118 Safari/537.36 Edg/131.0.4992.118; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.1981.4 Mobile Safari/537.36; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.211.19 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.194.58 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/19.5 Mobile/15E148 Safari/604.1; compatible; GPTBot/1.2; +https://openai.com/gptbot

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.860.219 Safari/537.36 Edg/134.0.860.219; compatible; Applebot/0.1; +http://www.apple.com/go/applebot

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (X11; Linux x86_64; rv:105.15) Gecko/20100101 Firefox/105.15; compatible; GrokBot/1.0; +https://x.ai/grokbot

Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.785.219 Mobile Safari/537.36; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Windows NT 10.0; rv:143.1) Gecko/20100101 Firefox/143.1; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:121.4) Gecko/20100101 Firefox/121.4; compatible; GrokBot/1.0; +https://x.ai/grokbot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.7528.96 Mobile Safari/537.36; compatible; GPTBot/1.2; +https://openai.com/gptbot

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.3339.43 Mobile Safari/537.36; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Chrome/109.0.818.225 Mobile Safari/537.36

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Chrome/85.0.7775.97 Mobile Safari/537.36

Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:132.12) Gecko/20100101 Firefox/132.12; compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.852.157 Safari/537.36; compatible; LinkedInBot/1.0; +http://www.linkedin.com

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.7026.241 Mobile Safari/537.36; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com) Chrome/131.0.4065.144 Mobile Safari/537.36

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.4916.67 Safari/537.36; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.6086.134 Safari/537.36 Edg/124.0.6086.134; compatible; Google-Extended/1.0; +http://www.google.com/bot.html

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.2852.228 Mobile Safari/537.36; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GrokBot/1.0; +https://x.ai/grokbot) Chrome/128.0.3160.32 Mobile Safari/537.36

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:149.13) Gecko/20100101 Firefox/149.13; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Applebot/0.1; +http://www.apple.com/go/applebot

Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; LinkedInBot/1.0; +http://www.linkedin.com) Chrome/131.0.1957.45 Mobile Safari/537.36

Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Safari/605.1.15; compatible; Discordbot/2.0; +https://discordapp.com

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) Chrome/124.0.7236.99 Mobile Safari/537.36

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.6707.118 Mobile Safari/537.36; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.5483.216 Mobile Safari/537.36; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GrokBot/1.0; +https://x.ai/grokbot) Chrome/133.0.223.210 Mobile Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Chrome/126.0.6180.224 Safari/537.36

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Chrome/126.0.5779.14 Safari/537.36

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Discordbot/2.0; +https://discordapp.com)

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot)

Mozilla/5.0 (iPhone; CPU iPhone OS 18_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.5 Mobile/15E148 Safari/604.1; compatible; Discordbot/2.0; +https://discordapp.com

Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; LinkedInBot/1.0; +http://www.linkedin.com) Chrome/126.0.7857.165 Mobile Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot) Chrome/109.0.4371.73 Mobile Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot) Chrome/136.0.4557.213 Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.6) Gecko/20100101 Firefox/78.6; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot

Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots) Version/17.6 Mobile/15E148 Safari/604.1

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; rv:133.0) Gecko/20100101 Firefox/133.0; compatible; ChatGPT-User/1.0; +https://openai.com/bot

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot) Chrome/151.0.8341.51 Safari/537.36

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.2296.210 Mobile Safari/537.36; compatible; GrokBot/1.0; +https://x.ai/grokbot

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/85.0.657.86 Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.2712.24 Safari/537.36 Edg/136.0.2712.24; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.1081.144 Safari/537.36; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/131.0.7595.20 Safari/537.36 Edg/131.0.7595.20

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Chrome/124.0.7846.145 Safari/537.36 Edg/124.0.7846.145

Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; LinkedInBot/1.0; +http://www.linkedin.com) Chrome/128.0.2276.79 Mobile Safari/537.36

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Version/19.0 Mobile/15E148 Safari/604.1

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.6273.204 Safari/537.36 Edg/130.0.6273.204; compatible; GPTBot/1.2; +https://openai.com/gptbot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; GPTBot/1.2; +https://openai.com/gptbot)

Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.2598.20 Safari/537.36; compatible; Bytespider; +https://zhanzhang.toutiao.com/

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Extended/1.0; +http://www.google.com/bot.html) Chrome/148.0.8932.35 Mobile Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 16_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Mobile/15E148 Safari/604.1; compatible; Bytespider; +https://zhanzhang.toutiao.com/

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GrokBot/1.0; +https://x.ai/grokbot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1 Safari/605.1.15; compatible; ChatGPT-User/1.0; +https://openai.com/bot

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6260.54 Safari/537.36; compatible; Applebot/0.1; +http://www.apple.com/go/applebot

Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots) Version/16.6 Mobile/15E148 Safari/604.1

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Chrome/148.0.3452.158 Safari/537.36 Edg/148.0.3452.158

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:143.9) Gecko/20100101 Firefox/143.9; compatible; LinkedInBot/1.0; +http://www.linkedin.com

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64; rv:121.6) Gecko/20100101 Firefox/121.6; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.1126.238 Mobile Safari/537.36; compatible; Discordbot/2.0; +https://discordapp.com

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.6637.197 Safari/537.36 Edg/122.0.6637.197; compatible; GrokBot/1.0; +https://x.ai/grokbot

Mozilla/5.0 (X11; Linux x86_64; rv:105.13) Gecko/20100101 Firefox/105.13; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot)

Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bytespider; +https://zhanzhang.toutiao.com/) Chrome/133.0.2313.35 Mobile Safari/537.36

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.6176.102 Mobile Safari/537.36; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler)

Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bytespider; +https://zhanzhang.toutiao.com/) Chrome/133.0.2224.194 Safari/537.36

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:149.7) Gecko/20100101 Firefox/149.7; compatible; Google-Extended/1.0; +http://www.google.com/bot.html

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.3090.210 Mobile Safari/537.36; compatible; ChatGPT-User/1.0; +https://openai.com/bot

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com)

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Chrome/91.0.5433.153 Mobile Safari/537.36

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.4332.186 Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.2775.12 Safari/537.36; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.741.185 Safari/537.36; compatible; Discordbot/2.0; +https://discordapp.com

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; Claude-User/1.0; +Claude-User@anthropic.com)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Chrome/136.0.1518.83 Mobile Safari/537.36

Mozilla/5.0 (compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots)

Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:143.8) Gecko/20100101 Firefox/143.8; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 (iPhone; CPU iPhone OS 18_4 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.377.124 Mobile Safari/537.36; compatible; Applebot/0.1; +http://www.apple.com/go/applebot

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.5937.176 Safari/537.36; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GrokBot/1.0; +https://x.ai/grokbot) Chrome/134.0.7079.243 Safari/537.36 Edg/134.0.7079.243

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:143.7) Gecko/20100101 Firefox/143.7; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) Chrome/118.0.5556.7 Mobile Safari/537.36

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.4312.22 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.1767.110 Mobile Safari/537.36; compatible; LinkedInBot/1.0; +http://www.linkedin.com

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler)

Mozilla/5.0 (compatible; LinkedInBot/1.0; +http://www.linkedin.com)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.13) Gecko/20100101 Firefox/121.13; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GrokBot/1.0; +https://x.ai/grokbot) Chrome/136.0.1609.238 Safari/537.36

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.1098.181 Safari/537.36; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.7183.226 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.8328.75 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.1158.199 Safari/537.36 Edg/114.0.1158.199; compatible; ChatGPT-User/1.0; +https://openai.com/bot

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/147.0.6145.91 Safari/537.36; compatible; GPTBot/1.2; +https://openai.com/gptbot

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:149.9) Gecko/20100101 Firefox/149.9; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.6507.198 Mobile Safari/537.36; compatible; Applebot/0.1; +http://www.apple.com/go/applebot

Mozilla/5.0 (compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php)

Mozilla/5.0 (compatible; Discordbot/2.0; +https://discordapp.com)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.1250.235 Mobile Safari/537.36; compatible; Google-Extended/1.0; +http://www.google.com/bot.html

Mozilla/5.0 (compatible; Bytespider; +https://zhanzhang.toutiao.com/)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.2108.19 Mobile Safari/537.36; compatible; GPTBot/1.2; +https://openai.com/gptbot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot) Chrome/131.0.1685.188 Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Discordbot/2.0; +https://discordapp.com

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 14; Pixel 8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.1848.119 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.8340.244 Safari/537.36; compatible; LinkedInBot/1.0; +http://www.linkedin.com

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.1126.222 Mobile Safari/537.36; compatible; ClaudeBot/1.0; +claudebot@anthropic.com

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:127.14) Gecko/20100101 Firefox/127.14; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:143.3) Gecko/20100101 Firefox/143.3; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user)

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.3754.89 Safari/537.36; compatible; GPTBot/1.2; +https://openai.com/gptbot

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/91.0.2234.205 Safari/537.36

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Chrome/133.0.752.134 Mobile Safari/537.36

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.1165.10 Safari/537.36; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler

Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.2037.73 Safari/537.36 Edg/114.0.2037.73; compatible; LinkedInBot/1.0; +http://www.linkedin.com

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.8732.74 Safari/537.36; compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot) Version/18.4 Mobile/15E148 Safari/604.1

Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.5990.204 Safari/537.36; compatible; Slackbot-LinkExpanding/1.0; +https://api.slack.com/robots

Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:143.0) Gecko/20100101 Firefox/143.0; compatible; facebookexternalhit/1.1; +http://www.facebook.com/externalhit_uatext.php

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.3168.112 Safari/537.36; compatible; Bytespider; +https://zhanzhang.toutiao.com/

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Version/19.5 Mobile/15E148 Safari/604.1

Mozilla/5.0 (Macintosh; Intel Mac OS X 14.5; rv:143.6) Gecko/20100101 Firefox/143.6; compatible; Discordbot/2.0; +https://discordapp.com

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 13; Pixel 7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) Chrome/148.0.7363.193 Mobile Safari/537.36

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Applebot/0.1; +http://www.apple.com/go/applebot)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/134.0.8512.166 Safari/537.36 Edg/134.0.8512.166; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.8171.162 Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot) Chrome/85.0.8300.176 Safari/537.36

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:127.8) Gecko/20100101 Firefox/127.8; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; ChatGPT-User/1.0; +https://openai.com/bot)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Bytespider; +https://zhanzhang.toutiao.com/)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.8378.222 Safari/537.36 Edg/126.0.8378.222; compatible; GPTBot/1.4; +https://openai.com/gptbot

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (compatible; GrokBot/1.0; +https://x.ai/grokbot)

Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:128.10) Gecko/20100101 Firefox/128.10; compatible; Discordbot/2.0; +https://discordapp.com

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user) Chrome/118.0.2246.25 Safari/537.36 Edg/118.0.2246.25

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot) Chrome/148.0.2021.20 Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.3919.178 Safari/537.36; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.4; +https://openai.com/gptbot) Chrome/148.0.1735.158 Mobile Safari/537.36

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.6929.87 Mobile Safari/537.36; compatible; ChatGPT-User/1.0; +https://openai.com/bot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Linux; Android 15; SM-S918B) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) Chrome/109.0.3241.21 Mobile Safari/537.36

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (X11; Ubuntu; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-User/1.0; +Claude-User@anthropic.com) Chrome/148.0.3976.71 Safari/537.36

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 13_6_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.0 Safari/605.1.15; compatible; Claude-User/1.0; +Claude-User@anthropic.com

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.2937.79 Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_5) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; OAI-SearchBot/1.4; robots.txt; +https://openai.com/searchbot) Version/17.5 Safari/605.1.15

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.7 Mobile/15E148 Safari/604.1; compatible; ChatGPT-User/1.0; +https://openai.com/bot

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Chrome/133.0.732.182 Safari/537.36

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +claudebot@anthropic.com

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.137.158 Mobile Safari/537.36; compatible; Claude-User/1.0; +https://www.anthropic.com/claude-user

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.7 Mobile/15E148 Safari/604.1; compatible; meta-externalagent/1.1; +https://developers.facebook.com/docs/sharing/webmasters/crawler

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Windows NT 11.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot) Chrome/148.0.7467.192 Safari/537.36

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GrokBot/1.0; +https://x.ai/grokbot)

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Bytespider; +https://zhanzhang.toutiao.com/

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Claude-SearchBot/1.0; +https://www.anthropic.com/claude-searchbot) Chrome/130.0.7044.228 Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.2; +https://openai.com/gptbot)

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (Linux; Android 12; Pixel 6) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user) Chrome/126.0.3654.143 Mobile Safari/537.36

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (Macintosh; Intel Mac OS X 14_6_1) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/19.2 Safari/605.1.15; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Google-Extended/1.0; +http://www.google.com/bot.html

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.7977.82 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/152.0.7977.64 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/152.0.7977.64 Safari/537.36

Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (HTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)

Mozilla/5.0 (compatible; fg-test-classifier/0.1; +https://fowlguard.com/bot)

Jetpack/12.0; WordPress/6.4; http://site19545037.com

UmbracoFingerprinter/1.0 (respects robots.txt)

LohiSoftBot/1.0 (+https://lohisoft.com/bot)

Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)

Mozilla/5.0 (compatible; srchs-research-bot/0.1; public small-business domain research; contact: smashcarlito@gmail.com)

checker-fast/1.0 (+https://example.org)

Mozilla/5.0 (compatible; DefaultsExposed/1.0; +https://defaults.exposed/headers)

Jetpack/12.1; WordPress/6.3; http://site60098938.com

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

agentcensus-crawler/1.0 (+https://agentcensus.dev/crawler)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.1; +https://openai.com/gptbot

Jetpack/12.5; WordPress/6.2; http://site41526053.com

HUNT-Bot/0.1 (+https://github.com/hunt-discovery; discovery engine)

Mozilla/5.0 (compatible; ForestEngine/1.0; +https://forestengine.net/)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/151.0.7922.173 Safari/537.36

Mozilla/5.0 (compatible; AwarioBot/1.0; +https://awario.com/bots.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.173 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Mozilla/5.0 (compatible; getdomaindata/1.0; +https://getdomaindata.com/bot)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot

Mozilla/5.0 (compatible; osentix-crawler/1.0; +https://osentix.com/bot)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 DataIndex.pro

Scrapy is an open-source Python framework used to build web crawlers and data-extraction applications.
Scrapy/2.17.0 (+https://scrapy.org)

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-webindexer/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.137 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

LohiSoftBot/1.0 (+message@lohisoft.com)

AgentTrustBot/1.0 (+https://agenttru.st/crawler; purpose=AI Agent Discovery; contact=admin@agenttru.st)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.7922.108 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/151.0.7922.108 Safari/537.36

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.0; +https://openai.com/gptbot)

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/144.0.0.0 Safari/537.36 Edg/144.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Jetpack/13.0; WordPress/6.4; http://site90475078.com

DomainsBot provides domain and business intelligence; Pandalytics gathers public web data for market and domain analysis.
Pandalytics/2.0 (+https://domainsbot.com/network-crawler-pandalytics-kdc68dhsbd/)

Jetpack by WordPress.com (Jetpack 12.5; WordPress 6.1)

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (compatible; Amzn-SearchBot/1.0; +https://developer.amazon.com/support/amazonbot)

Jetpack by WordPress.com

Jetpack/12.1; WordPress/6.3; http://site61741366.com

Jetpack by WordPress.com (Jetpack 12.0; WordPress 6.3)

BuiltWith identifies the technologies used by websites and produces web technology, market, and lead intelligence.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.4; rb.gy/xprgqj; 2fa6014a47) Chrome/124.0.0.0 Safari/537.36

serpstatbot/2.1 (advanced backlink tracking bot; https://serpstatbot.com/; abuse@serpstatbot.com)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/150.0.7871.186 Safari/537.36

Mozilla/5.0 (compatible; SERankingBacklinksBot/1.0; +https://seranking.com/backlinks-crawler)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.7871.186 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

understory/0.1 (CT research; https://github.com/0x4b) +contact-via-repo

siteradar/0.1 contact: ops@yyai.studio

LeakIX scans internet-facing services and indexes exposed systems, service banners, leaks, and security findings.
Mozilla/5.0 (l9scan/2.0.933323e2033313e29393e2933313; +https://leakix.net)

Jetpack by WordPress.com (Jetpack 12.1; WordPress 6.4)

WordPress.com; https://wordpress.com

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36 (Silvy X Ran; +https://silvyxran.love; +https://silver.inc)

Mozilla/5.0 (compatible; rust_sniffer/0.1; +https://github.com/)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Googlebot/2.1 (+http://www.google.com/bot.html)

ByteDance operates content platforms including TikTok; Bytespider collects public web content for ByteDance services.
Mozilla/5.0 (Linux; Android 5.0) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; Bytespider; spider-feedback@bytedance.com)

Mozilla/5.0 (compatible; DuckDuckBot/1.1; +http://duckduckgo.com/duckduckbot.html)

Mozilla/5.0 (compatible; GoogleOther; +https://developers.google.com/search/)

ucp-crawler/0.1 (+https://github.com/huckleberry-inc/ucp-crawler)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 AICrawler.store

Mozilla/5.0 (Windows NT 10.0.0; Win64; x64; ) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.6367.63 Chrome/124.0.6367.63 Not-A.Brand/99 Safari/537.36

Mozilla/5.0 (compatible; MJ12bot/v2.0.5; http://mj12bot.com/)

Mozilla/5.0 (compatible; jscrawler/0.1; +https://github.com/)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.7871.128 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Microsoft Bing operates bingbot to discover and index public pages for Bing search and related Microsoft services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/136.0.0.0 Safari/537.36 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36

Mozilla/5.0 (compatible; domains-distillery/1.0; +https://crawler.validus.ai)

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 Edg/145.0.0.0 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Nessus SOAP v0.0.1 (Nessus.org)

UMD-gRPC-Research (academic gRPC measurement; https://www.cs.umd.edu/~stekin/scan/; contact: baddatalab@umd.edu)

httpx - Open-source project (github.com/projectdiscovery/httpx)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.7871.124 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.7871.114 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Mozilla/5.0 (compatible; heritrix/3.14.2-SNAPSHOT-2026-04-13T06:21:22Z +https://www.image-meta.com)

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36 (compatible; meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler))

Jetpack/12.1; WordPress/6.1; http://site52012413.com

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/150.0.7871.46 Safari/537.36

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko, Yokohama Institute of Information Security https://www.iisec.ac.jp) Chrome/124.0.0.0 Safari/537.36

Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041117 Firefox/1.0 (Debian package 1.0-2.0.0.45.linspire0.4)

HTC-ST7377/1.59.502.3 (67150) Opera/9.50 (Windows NT 5.1; U; en) UP.Link/6.3.1.17.0

Mozilla/5.0 (compatible; FlowIQLabsBot/1.0; +https://flowiq-labs.com/scanning-info)

Mozilla/5.0 (compatible; NetcraftSurveyAgent/1.0; +info@netcraft.com)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36 +https://forestengine.net/#opt-out

Jetpack/12.5; WordPress/6.4; http://site74961882.com

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +https://www.google.com/bot.html) Safari/537.36

Jetpack by WordPress.com (Jetpack 12.1; WordPress 6.2)

Mozilla/5.0 (compatible; Infrawatch/1.0; +https://infrawat.ch/)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.7871.46 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/149.0.7827.200 Safari/537.36

Mozilla/5.0 (iPhone; CPU iPhone OS 17_5 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.5 Mobile/15E148 Safari/604.1 CibraxScanner/1.0 (+https://cibrax.com/scanner)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/149.0.7827.200 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Microsoft Bing operates bingbot to discover and index public pages for Bing search and related Microsoft services.
Mozilla/5.0 (compatible; Bingbot/2.0; +http://www.bing.com/bingbot.htm)

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

Wget/1.21.4 (HelpMeEscapeFromBelarus@proton.me)

Mozilla/5.0 (compatible; Exabot/3.0; http://www.exabot.com/go/robot)

FAST-WebCrawler/3.8 (crawler at trd dot overture dot com; http://www.alltheweb.com/help/webmaster/crawler)

Mozilla/5.0 (SymbianOS/9.2; U; Series60/3.1 NokiaE90-1/07.24.0.3; Profile/MIDP-2.0 Configuration/CLDC-1.1 ) AppleWebKit/413 (KHTML, like Gecko) Safari/413 UP.Link/6.2.3.18.0

Mozilla/5.0 (compatible; alexa site audit/1.0; +http://www.alexa.com/help/webmasters; )

MOT-V177/0.1.75 UP.Browser/6.2.3.9.c.12 (GUI) MMP/2.0 UP.Link/6.3.1.13.0

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (compatible; Googlebot/2.1; http://www.google.com/bot.html)

Mozilla/5.0 (compatible; archive.org_bot; Wayback Machine Live Record; +http://archive.org/details/archive.org_bot)

msnbot/1.0 ( http://search.msn.com/msnbot.htm)

SonyEricssonW810i/R4EA Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.0.0.0

grub-client-1.5.3; (grub-client-1.5.3; Crawl your own stuff with http://grub.org)

Mozilla/5.0 (compatible; Yahoo! Slurp China; http://misc.yahoo.com.cn/help.html)

MOT-V9mm/00.62 UP.Browser/6.2.3.4.c.1.123 (GUI) MMP/2.0

WebZIP/3.5 (http://www.spidersoft.com)

facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (iPhone; U; CPU iPhone OS) (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html)

Mozilla/5.0 (Linux; Android 5.1.1; vivo Y31L.RastaMod™_Version) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.143 Mobile Safari/537.36

Bloglines/3.1 (http://www.bloglines.com)

Mozilla/5.0 (X11; Linux 3.8-6.dmz.1-liquorix-686) KHTML/4.8.4 (like Gecko) Konqueror/4.8

Mozilla/5.0 (compatible; GeniusDataBot/1.0; +https://geniusdata.com)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html)

Mozilla/5.0 (compatible; SemrushBot/7~bl; +http://www.semrush.com/bot.html)

Mozilla/5.0 (compatible; DomainScores/2.1; +https://www.domainscores.net/headers)

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot) Chrome/119.0.6045.214 Safari/537.36

TLS tester from https://testssl.sh/

BlackBerry7520/4.0.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/5.0.3.3 UP.Link/5.1.2.12 (Google WAP Proxy/1.0)

Gulper Web Bot 0.2.4 (www.ecsl.cs.sunysb.edu/~maxim/cgi-bin/Link/GulperBot)

Mozilla/5.0 (compatible; archive.org_bot +http://www.archive.org/details/archive.org_bot)

NokiaN70-1/5.0609.2.0.1 Series60/2.8 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.1.13.0

Mozilla/5.0 (Linux; U; Android 8.1.0; zh-CN; EML-AL00 Build/HUAWEIEML-AL00) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/57.0.2987.108 baidu.sogo.uc.UCBrowser/11.9.4.974 UWS/2.13.1.48 Mobile Safari/537.36 AliApp(DingTalk/4.5.11) com.alibaba.android.rimet/10487439 Channel/227200 language/zh-CN

MOT-L7v/08.B7.5DR MIB/2.2.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.0.0.0

SEC-SGHX210/1.0 UP.Link/6.3.1.13.0

Yandex operates a search engine; YandexBot discovers and indexes public pages for its search services.
Mozilla/5.0 (compatible; YandexNews/4.0; +http://yandex.com/bots)

msnbot/0.11 ( http://search.msn.com/msnbot.htm)

everyfeed-spider/2.0 (http://www.everyfeed.com)

Gregarius/0.5.2 ( http://devlog.gregarius.net/docs/ua)

msnbot/1.1 ( http://search.msn.com/msnbot.htm)

BlackBerry9530/4.7.0.167 Profile/MIDP-2.0 Configuration/CLDC-1.1 VendorID/102 UP.Link/6.3.1.20.0

msnbot-media/1.1 ( http://search.msn.com/msnbot.htm)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; http://www.google.com/bot.html)

Googlebot/2.1 ( http://www.googlebot.com/bot.html)

SonyEricssonK800i/R1CB Browser/NetFront/3.3 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.0.0.0

Mozilla/5.0 (Linux; Android 8.1.0; LM-Q710.FG) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.101 Mobile Safari/537.36

SonyEricssonZ800/R1Y Browser/SEMC-Browser/4.1 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Link/6.3.0.0.0

Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) baidu.sogo.uc.Chrome/36.0.1985.125 Safari/537.36

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
AdsBot-Google ( http://www.google.com/adsbot.html)

Gaisbot/3.0 (robot@gais.cs.ccu.edu.tw; http://gais.cs.ccu.edu.tw/robot.php)

Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)

check-host.cc Intel Platform (https://check-host.cc/about/scan-infra; abuse@check-host.cc)

Mozilla/5.0 (compatible; SparixEmailScraper/1.0; +https://sparix.ai)

Mozilla/5.0 (compatible; wpbot/1.4; +https://forms.gle/ajBaxygz9jSR8p8G9)

2ip bot/1.1 (+https://2ip.io/bot/)

Mozilla/5.0 (compatible; Dataprovider.com)

Mozilla/5.0 (compatible; MJ12bot/v1.4.8; http://mj12bot.com/)

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 AISearchIndex.space

Mozilla/5.0 (compatible; Nmap Scripting Engine; https://nmap.org/book/nse.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Googlebot-Image/1.0

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/148.0.7778.96 Safari/537.36

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

BuiltWith identifies the technologies used by websites and produces web technology, market, and lead intelligence.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; BuiltWith/1.4; rb.gy/xprgqj) Chrome/124.0.0.0 Safari/537.36

Microsoft Bing operates bingbot to discover and index public pages for Bing search and related Microsoft services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/136.0.0.0 Safari/537.36

Microsoft Bing operates bingbot to discover and index public pages for Bing search and related Microsoft services.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36

Example.com is a reserved documentation domain, so a scanner using it in its user agent does not identify its real operator.
Mozilla/5.0 (compatible; CMS-Checker/1.0; +https://example.com)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (Linux; Android 13; Pixel 4a (5G) Build/TQ2A.230505.002; wv) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/112.0.5615.136 Mobile Safari/537.36 GoogleApp/14.16.27.29.arm64 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.5 Safari/605.1.15 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (Android 13; Mobile; rv:109.0) Gecko/112.0 Firefox/112.0 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Google Code is a legacy Google developer site; this agent identifies an application running on Google App Engine.
Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 AppEngine-Google; (+http://code.google.com/appengine; appid: s~virustotalcloud)

Yandex operates a search engine; YandexBot discovers and indexes public pages for its search services.
Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; OAI-SearchBot/1.3; +https://openai.com/searchbot)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Microsoft Bing operates bingbot to discover and index public pages for Bing search and related Microsoft services.
Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ChatGPT-User/1.0; +https://openai.com/bot)

Baidu is a major Chinese search engine; Baiduspider crawls pages for its web search index.
Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.3; +https://openai.com/gptbot)

xAI develops Grok; xAI-SearchBot discovers public web content for search and AI-generated answers.
Mozilla/5.0 (compatible; xAI-SearchBot/1.0; +https://x.ai)

Google Cloud provides cloud computing and Vertex AI; Google-CloudVertexBot retrieves pages for Vertex AI agent features.
Mozilla/5.0 (compatible; Google-CloudVertexBot; +https://cloud.google.com/vertex-ai-bot)

DeepSeek develops large language models; DeepSeekBot retrieves public web content for its search and AI services.
Mozilla/5.0 (compatible; DeepSeekBot/1.0; +https://www.deepseek.com/bot)

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 (compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)

CheckMark Network crawls the web to support trademark, domain, ecommerce, image, and brand-monitoring services.
CheckMarkNetwork/1.0 (+http://www.checkmarknetwork.com/spider.html)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Googlebot-News

CyberConvoy Scout performs identifiable internet scanning to discover publicly reachable services and security exposures.
Mozilla/5.0 (compatible; CyberConvoyScout/1.0; +https://scout.cyberconvoy.co)

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/148.0.7778.96 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

DomainsBot provides domain and business intelligence; Pandalytics gathers public web data for market and domain analysis.
Pandalytics/2.0 (https://domainsbot.com/pandalytics/)

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)

Driftnet operates this domain to discover and measure publicly exposed network services without attempting to log in.
Mozilla/5.0 (compatible; InternetMeasurement/1.0; +https://internet-measurement.com/)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot

ByteDance operates content platforms including TikTok; Bytespider collects public web content for ByteDance services.
Mozilla/5.0 (compatible; Bytespider; spider-feedback@bytedance.com)

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.2; +https://openai.com/gptbot

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 (compatible; Claude-Web/1.0; +https://www.anthropic.com)

Amazon's developer portal hosts the policy for Amazonbot, a crawler that supports Amazon search and AI services.
Mozilla/5.0 (compatible; Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)

OpenAI develops ChatGPT; its agents retrieve public pages for search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ChatGPT-User/1.0; +https://openai.com/bot

Meta's developer platform documents meta-externalagent, which fetches public web content for Meta products and AI systems.
meta-externalagent/1.1 (+https://developers.facebook.com/docs/sharing/webmasters/crawler)

Apple operates Applebot to index public web content for products such as Siri, Spotlight, and Safari search features.
Mozilla/5.0 (compatible; Applebot/0.1; +http://www.apple.com/go/applebot)

Cohere develops enterprise language and retrieval models; its crawler collects public content for AI products and models.
Mozilla/5.0 (compatible; cohere-ai/1.0; +https://cohere.com)

Perplexity is an AI answer engine whose agents crawl or fetch public pages for search results and user questions.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; Perplexity-User/1.0; +https://perplexity.ai/perplexity-user

Google operates crawlers including Googlebot to discover and index public content for Search and related services.
Mozilla/5.0 (compatible; Google-Extended/1.0; +http://www.google.com/bot.html)

Anthropic develops Claude; its crawlers retrieve public web content for Claude search, user-requested browsing, and model development.
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; ClaudeBot/1.0; +mailto:support@anthropic.com

Common Crawl operates CCBot and publishes an open repository of web crawl data for research and analysis.
CCBot/2.0 (https://commoncrawl.org/faq/)

Recorded Future collects and analyzes internet data to produce threat intelligence; its crawler inventories public assets.
RecordedFuture Global Inventory Crawler

Censys continuously scans public internet services and certificates to build an internet-wide asset and exposure index.
Mozilla/5.0 (compatible; CensysInspect/1.1; +https://about.censys.io/)

IPIP.net's security scanner probes public services and collects HTTP banner information for internet asset analysis.
HTTP Banner Detection (https://security.ipip.net)

Sogou is a Chinese search service; Sogou web spider discovers and indexes public pages for its search engine.
Sogou web spider/4.0(+http://www.sogou.com/docs/help/webmasters.htm#07)

VisionHeight performs ethical internet scanning to map adversary infrastructure and provide pre-attack threat intelligence.
visionheight.com/scan Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) Chrome/126.0.0.0 Safari/537.36

Nokia Deepfield's Genome crawler maps public IP addresses, applications, and services to provide internet security context.
Mozilla/5.0 (compatible; GenomeCrawlerd/1.0; +https://www.nokia.com/genomecrawler)

Modat combines large-scale internet scanning with infrastructure context to identify exposures and emerging cyber threats.
Mozilla/5.0 (compatible; ModatScanner/1.2; +https://modat.io/)