[{"content":"Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.\nThese checks evaluate whether agents can find and navigate the site\u0026rsquo;s documentation content. This includes whether the site provides an llms.txt file, whether that file is useful to agents, and whether documentation pages include signals that direct agents to discovery resources.\nLocation Discovery #The llmstxt.org proposal specifies that llms.txt should be at the root path (/llms.txt), mirroring robots.txt and sitemap.xml. In practice, the location varies significantly across sites:\nSite Root /llms.txt /docs/llms.txt Notes MongoDB 200 200 Both locations, different content Neon 200 200 Both locations Stripe 200 301 -\u0026gt; docs.stripe.com Root + docs subdomain Vercel 200 308 -\u0026gt; root Root only, /docs redirects React 200 \u0026ndash; Root only GitHub Docs 200 \u0026ndash; Root only Claude Code 302 -\u0026gt; product page 200 /docs only; root is not docs Anthropic (old) 301 -\u0026gt; 404 \u0026ndash; Moved domain, redirect breaks The proposal does not address whether sites should serve llms.txt at subpaths, or whether a site with docs at /docs/ should place it at /docs/llms.txt vs /llms.txt. In practice, both patterns exist. Implementations should check multiple candidate locations.\nDiscovery algorithm: Given a base URL, check for llms.txt at:\n{base_url}/llms.txt (the exact URL the user provided, plus llms.txt) {origin}/llms.txt (site root, per the proposal) {origin}/docs/llms.txt (common docs subpath) Where {origin} is the scheme + host of the base URL, and {base_url} is the full URL the user provided (which might be https://example.com/docs or https://example.com or https://docs.example.com). Duplicate URLs are deduplicated before checking.\nFor each location, record whether llms.txt exists and whether the response involved a redirect (and if so, what kind). All subsequent llms.txt checks run against every discovered llms.txt file.\nllms-txt-exists # What it checks: Whether llms.txt is discoverable at any of the candidate locations described above. Why it matters: llms.txt was the single most effective discovery mechanism observed. When agents found one, it fundamentally changed their ability to navigate a documentation site. Agents don\u0026rsquo;t know to look for llms.txt by default, but when pointed at one, they treat it as a primary navigation resource. Result levels: Pass: llms.txt exists at one or more candidate locations, returning 200 with text content (direct or after same-host redirect). Warn: llms.txt exists but is only reachable via cross-host redirect (agents may not follow it). Fail: llms.txt not found at any candidate location. Recommended action: Warn: Serve llms.txt directly from the same host as your documentation, or use a same-host redirect. Cross-host redirects are not followed by some agents. Fail: Create an llms.txt file at your site root containing an H1 title, a blockquote summary, and markdown links to your key documentation pages. This is the single highest-impact improvement for agent access. Automation: Full. Report details: List all candidate URLs checked and their status (200, 404, redirect chain). When multiple locations return llms.txt, note whether they serve the same or different content. llms-txt-valid # What it checks: Whether the llms.txt follows the structure described in the llmstxt.org proposal. The proposal specifies: An H1 with the project/site name. A blockquote with a short summary. H2-delimited sections containing markdown link lists. Each link entry: [name](url) optionally followed by : description. An optional H2 \u0026ldquo;Optional\u0026rdquo; section for secondary content. Optional companion file llms-full.txt with complete content. Why it matters: A well-structured llms.txt gives agents a reliable map of the documentation. Inconsistent implementations reduce its value. That said, even a non-standard llms.txt that contains useful links is better than nothing. Result levels: Pass: Follows the proposed structure with H1, summary blockquote, and heading-delimited link sections. Warn: Contains parseable markdown links but doesn\u0026rsquo;t follow the proposed structure (still useful, just non-standard). Fail: Exists but contains no parseable links, or is empty. Recommended action: Warn: Add an H1 title as the first line and a blockquote summary (lines starting with \u0026gt;) to improve agent parsing. Fail: Add links in [name](url): description format under heading-delimited sections. Automation: Full. Checks in detail: H1 present (first line starts with # ). Blockquote summary present (line starting with \u0026gt; ). At least one heading-delimited section with markdown links. Links follow [name](url) format. Optional: check for llms-full.txt companion file. Notes on heading levels: The llmstxt.org proposal specifies H2 (##) for section delimiters. In practice, some implementations (notably MongoDB) use H1 (#) for sections instead. Implementations should accept any heading level for section delimiters when evaluating structure. The important thing is that sections exist and contain parseable links, not that they use a specific heading level. llms-txt-links-resolve # What it checks: Whether the URLs listed in llms.txt resolve to the content they promise. Why it matters: A stale llms.txt with broken links is worse than no llms.txt at all. It sends agents down dead ends with high confidence. Result levels: Pass: All links resolve (200, following same-host redirects) with no soft 404s or representation mismatches. Warn: \u0026gt;90% of links resolve. Fail: \u0026lt;=90% of links resolve. Recommended action: Audit and fix or remove broken URLs. A stale llms.txt with broken links is worse than no llms.txt at all because it sends agents down dead ends with high confidence. Automation: Full. Verification must go beyond status codes: apply soft-404 heuristics (see http-status-codes) to the response body, and verify the content type for links that promise markdown. A bare status check passes broken links; in one observed production case, a generated catalog\u0026rsquo;s .md links all soft-404ed as HTML SPA shells while returning 200 (see markdown-link-portability). Notes: Requires making HTTP requests to each URL. For large files, implementations may choose to test a random subset rather than every link. Links in llms.txt should be absolute URLs: the file is copied, cached, and aggregated by tools that drop its source URL, so relative links share the portability failure modes described in markdown-link-portability. The progressive disclosure design principles make the same recommendation for links between index levels. llms-txt-size # What it checks: The character count of the llms.txt file, and whether it exceeds the truncation limits of known agent web fetch pipelines.\nWhy it matters: An llms.txt that exceeds an agent\u0026rsquo;s truncation limit defeats its own purpose. The agent sees only a fraction of the index and may miss the section it needs entirely. This is the same truncation problem that affects documentation pages, but arguably worse because llms.txt is supposed to be the solution to discovery.\nReal-world sizes vary enormously:\nSite Size Links Notes MongoDB /docs/llms.txt 4.56 MB 21,891 Every version of every product Vercel 287 KB ~3,000 Single file Stripe 89 KB ~1,000 Single file Neon 75 KB ~600 Points to .md URLs React 14 KB ~150 Single file Claude Code 11 KB ~60 Small, focused GitHub Docs 2 KB ~30 Small index MongoDB /llms.txt (root) 1.5 KB 6 Top-level index only Claude Code\u0026rsquo;s web fetch pipeline truncates at ~100KB. A 4.56MB file means the agent sees roughly 2% of it. Even Vercel\u0026rsquo;s 287KB file would be heavily truncated. Only the files under ~100KB are reliably consumable in their entirety by current agent implementations.\nResult levels:\nPass: Under 50,000 characters (fits comfortably within all known truncation limits, even accounting for overhead). Warn: Between 50,000 and 100,000 characters (fits within Claude Code\u0026rsquo;s limit but may not fit others; consider splitting). Fail: Over 100,000 characters (will be truncated by Claude Code and likely all other agent platforms). Recommended action:\nWarn: If the file grows further, split into nested llms.txt files with a root index under 50,000 characters. Fail: Split into a root index linking to section-level llms.txt files, each under 50,000 characters. See Progressive Disclosure for Large Documentation Sets below. Automation: Full.\nllms-txt-links-markdown # What it checks: Whether the URLs in llms.txt point to markdown content (.md extension in the URL, or response with Content-Type: text/markdown). Why it matters: Markdown content is dramatically more useful to agents than HTML. An llms.txt that points agents to HTML pages misses an opportunity to deliver content in the most agent-friendly format. The best implementations (like Neon\u0026rsquo;s) point to .md URLs that serve clean markdown directly. Result levels: Pass: All or most links point to markdown content. Warn: Links point to HTML, but markdown versions are available (detected by trying .md variants of the URLs). Fail: Links point to HTML and no markdown alternatives are detected. Recommended action: Update llms.txt links to use .md URL variants so agents receive markdown instead of converted HTML. Automation: Full. Progressive Disclosure for Large Documentation Sets #The llmstxt.org proposal does not address what to do when a documentation site is too large for a single llms.txt file to fit within agent truncation limits. In practice, large documentation sets (like MongoDB\u0026rsquo;s, with 185 products/versions and 21,891 links) produce llms.txt files that are orders of magnitude beyond what any current agent can consume in a single fetch.\nWho Actually Uses llms.txt? #The original framing of llms.txt drew analogies to robots.txt and sitemap.xml, suggesting it would serve AI crawlers gathering training data. The evidence shows this hasn\u0026rsquo;t happened:\nAn audit of 1,000 domains over 30 days found zero visits to llms.txt from GPTBot, ClaudeBot, or PerplexityBot (Longato, August 2025). A 90-day study tracking 62,100+ AI bot visits found only 84 requests (0.1%) to /llms.txt, roughly 3x fewer visits than an average content page (OtterlyAI GEO Study). John Mueller from Google stated directly: \u0026ldquo;no AI system currently uses llms.txt.\u0026rdquo; Training crawlers don\u0026rsquo;t use llms.txt because they have their own discovery mechanisms (sitemaps, link following, pre-built datasets) and probing /llms.txt on every domain would waste crawl budget for an unestablished standard.\nThe real consumers of llms.txt are agents in real-time workflows: a developer\u0026rsquo;s coding assistant fetching documentation to verify an API pattern, an agent following a directive on a docs page that points it to llms.txt, or a user explicitly handing their agent an llms.txt URL as a discovery starting point. These are fetch-once, use-now interactions subject to the truncation limits of web fetch pipelines.\nThis distinction matters for our recommendation. A progressive disclosure pattern that splits llms.txt into nested files has no practical impact on crawler consumption (since crawlers aren\u0026rsquo;t consuming it). It directly benefits the agent use case, which is where llms.txt actually provides value today.\nRecommendation #We recommend a nested llms.txt pattern for progressive disclosure:\nStructure #A root llms.txt serves as a table of contents, listing the major sections of the documentation with links to section-level llms.txt files. Each section-level file contains the actual page links for that section.\n# MongoDB Documentation \u0026gt; MongoDB is the leading document database. This index covers all MongoDB \u0026gt; products, drivers, and tools documentation. ## Products - [Atlas](https://www.mongodb.com/docs/atlas/llms.txt): MongoDB Atlas cloud database - [Atlas CLI](https://www.mongodb.com/docs/atlas-cli/llms.txt): Command-line interface for Atlas - [Compass](https://www.mongodb.com/docs/compass/llms.txt): GUI for MongoDB - [MongoDB Server](https://www.mongodb.com/docs/manual/llms.txt): Server documentation ## Drivers - [Python Driver](https://www.mongodb.com/docs/drivers/pymongo/llms.txt): PyMongo driver - [Node.js Driver](https://www.mongodb.com/docs/drivers/node/llms.txt): Node.js driver - [Java Driver](https://www.mongodb.com/docs/drivers/java/llms.txt): Java sync and reactive drivers Each linked llms.txt then contains the actual page listings for that product or driver, scoped to the current version (or with a small number of version variants).\nDesign Principles # The root llms.txt should fit in a single agent fetch. Target under 50,000 characters. This is the entry point that agents will discover first, and it must be fully consumable. It should contain enough descriptive context for an agent to identify which section-level file to fetch next.\nSection-level files should also fit in a single agent fetch. If a section is still too large (e.g., a product with hundreds of pages across many versions), consider further nesting or limiting the index to the current version only.\nVersion sprawl is the primary size driver. The MongoDB /docs/llms.txt lists every version of every product. Linking to every historical version in the index provides diminishing returns for agents, who almost always want the current version. Historical versions could be listed in a separate llms-versions.txt or under the \u0026ldquo;Optional\u0026rdquo; H2 section that the proposal already defines for secondary content.\nLinks between levels should use absolute URLs. An agent following a link from root llms.txt to a section llms.txt needs to resolve it without ambiguity.\nEach llms.txt should be self-describing. Include the H1 and blockquote summary at every level so an agent landing on a section-level file (via direct URL from training data, for example) has enough context to understand what it\u0026rsquo;s looking at.\nCompatibility Note #This nested pattern is a recommendation from this spec, not part of the llmstxt.org proposal as of February 2026. It is fully compatible with the existing proposal (which doesn\u0026rsquo;t prohibit linking to other llms.txt files) but would benefit from formal standardization. The proposal\u0026rsquo;s existing \u0026ldquo;Optional\u0026rdquo; H2 section could be leveraged for secondary/versioned content, but the nesting pattern goes further by distributing content across multiple files.\nllms-txt-directive-html # What it checks: Whether the HTML version of documentation pages includes a directive, visible to agents but not necessarily to human readers, pointing to llms.txt or another discovery resource. Why it matters: Agents that fetch rendered HTML pages have no built-in way to discover that a documentation index exists at /llms.txt or that markdown versions of pages may be available. An in-page directive serves as an agent \u0026ldquo;You Are Here\u0026rdquo; marker that points them to the index. The directive can be visually hidden (e.g., using a CSS clip-rect technique or sr-only class) as long as it remains in the DOM and survives HTML-to-markdown conversion. Avoid display: none, which some converters strip. The directive should be present in server-rendered HTML; avoid relying solely on client-side JavaScript injection, since most agents fetch pages without executing JS. Detection considerations: Implementations must distinguish intentional agent-facing directives from incidental mentions of llms.txt. Navigation items (e.g., sidebar links to a page about llms.txt), JSON-LD metadata, \u0026lt;script\u0026gt; blocks, and page content that merely discusses llms.txt as a feature do not count as directives. A directive is a standalone element in the page content area whose purpose is to tell agents where to find the documentation index. Result levels: Pass: A directive pointing to llms.txt (or equivalent index) is present in the HTML DOM of all (or nearly all) documentation pages, ideally near the top of the content. Warn: A directive exists in the HTML of some pages but is missing from others, or is present but buried deep in the page (past 50% of content, where it may be past truncation). Fail: No agent-facing directive detected in the HTML of any tested page. Recommended action: Warn: Ensure the directive appears near the top of every documentation page\u0026rsquo;s HTML, not just some. Fail: Add a visually-hidden element near the top of each page (e.g., a \u0026lt;div\u0026gt; with CSS clip-rect) containing text like \u0026ldquo;For AI agents: a documentation index is available at /llms.txt\u0026rdquo; and, if applicable, a note that markdown versions are available. Automation: Heuristic. Search the page HTML for patterns like links to llms.txt, phrases like \u0026ldquo;documentation index\u0026rdquo;, or directives near the top of the content area. Check both visible text and visually-hidden elements. Exclude matches in navigation, metadata, script blocks, and content that discusses llms.txt as a topic rather than directing agents to it. llms-txt-directive-md # What it checks: Whether the markdown version of documentation pages includes a directive pointing to llms.txt or another discovery resource. Why it matters: Agents that fetch markdown versions of pages (via .md URLs or content negotiation) benefit from a directive that points them to the documentation index. Anthropic\u0026rsquo;s Claude Code documentation (code.claude.com/docs, hosted on Mintlify) includes a blockquote at the top of every markdown page telling agents to fetch the documentation index at llms.txt. In practice, agents that encounter this directive may follow it to discover the full documentation index. It\u0026rsquo;s simple, low-effort, and has been observed to work in real agent workflows. Result levels: Pass: A directive pointing to llms.txt (or equivalent index) is present in the markdown of all (or nearly all) documentation pages, ideally near the top of the content. Warn: A directive exists in the markdown of some pages but is missing from others, or is present but buried deep in the page (past 50% of content, where it may be past truncation). Fail: No agent-facing directive detected in the markdown of any tested page. Recommended action: Warn: Ensure the directive appears near the top of every markdown page, not just some. Fail: Add a blockquote near the top of each markdown page (e.g., \u0026ldquo;\u0026gt; For the complete documentation index, see llms.txt\u0026rdquo;). Use an absolute URL; see markdown-link-portability for why relative links are fragile in served markdown. Automation: Heuristic. Fetch the markdown version of sampled pages (via .md URL or content negotiation) and search for patterns like links to llms.txt, phrases like \u0026ldquo;documentation index\u0026rdquo;, or blockquote directives near the top of the content. This check depends on markdown being available via markdown-url-support or content-negotiation; if neither passes, this check is skipped. ","date":null,"permalink":"https://agentdocsspec.com/spec/web/content-discoverability/","section":"Specifications","summary":"\u003cp\u003ePart of the \u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e. The \u003ca href=\"https://agentdocsspec.com/spec/web/#checks-summary\" target=\"_blank\" rel=\"noreferrer\"\u003eChecks Summary\u003c/a\u003e lists all checks with links to their definitions.\u003c/p\u003e\n\u003cp\u003eThese checks evaluate whether agents can find and navigate the site\u0026rsquo;s\ndocumentation content. This includes whether the site provides an \u003ccode\u003ellms.txt\u003c/code\u003e\nfile, whether that file is useful to agents, and whether documentation pages\ninclude signals that direct agents to discovery resources.\u003c/p\u003e\n\u003ch3 id=\"location-discovery\" class=\"relative group\"\u003eLocation Discovery \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#location-discovery\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cp\u003eThe \u003ca href=\"https://llmstxt.org\" target=\"_blank\" rel=\"noreferrer\"\u003ellmstxt.org proposal\u003c/a\u003e specifies that \u003ccode\u003ellms.txt\u003c/code\u003e\nshould be at the root path (\u003ccode\u003e/llms.txt\u003c/code\u003e), mirroring \u003ccode\u003erobots.txt\u003c/code\u003e and\n\u003ccode\u003esitemap.xml\u003c/code\u003e. In practice, the location varies significantly across sites:\u003c/p\u003e","title":"Category 1: Content Discoverability"},{"content":"Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.\nThese checks evaluate whether the site serves documentation in markdown format, which agents consume far more effectively than HTML.\nmarkdown-url-support # What it checks: Whether appending .md to documentation page URLs returns valid markdown content. Why it matters: Agents work dramatically better with markdown than HTML. The HTML-to-markdown conversion in web fetch pipelines is lossy and unpredictable. Sites that serve markdown directly bypass conversion issues entirely. However, agents don\u0026rsquo;t discover this pattern on their own; it needs to be signaled. Result levels: Pass: .md URLs return valid markdown with 200 status. Warn: Some pages support .md but not consistently. Fail: .md URLs return errors or HTML. Recommended action: Warn: Ensure all documentation pages serve markdown when .md is appended to the URL, not just some. Fail: Configure your docs platform to serve .md variants for all documentation pages. Automation: Full. Test against a sample of page URLs (from llms.txt, sitemap, or user-provided list). content-negotiation # What it checks: Whether the server responds to Accept: text/markdown with markdown content and an appropriate Content-Type header. Why it matters: Some agents (Claude Code, Cursor, OpenCode) send Accept: text/markdown as their preferred content type. If the server honors this, the agent gets clean markdown without needing to know about .md URL patterns. Most agents don\u0026rsquo;t request markdown, but the ones that do should get it. Result levels: Pass: Server returns markdown content with Content-Type: text/markdown when requested. Warn: Server returns markdown content but with incorrect Content-Type. Fail: Server ignores the Accept header and returns HTML regardless. Recommended action: Warn: Set the response Content-Type to text/markdown when serving markdown content. The correct header enables optimizations in some agent pipelines. Fail: Configure your server to honor Accept: text/markdown requests and return markdown content. Some agents (Claude Code, Cursor, OpenCode) request markdown this way. Automation: Full. ","date":null,"permalink":"https://agentdocsspec.com/spec/web/markdown-availability/","section":"Specifications","summary":"\u003cp\u003ePart of the \u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e. The \u003ca href=\"https://agentdocsspec.com/spec/web/#checks-summary\" target=\"_blank\" rel=\"noreferrer\"\u003eChecks Summary\u003c/a\u003e lists all checks with links to their definitions.\u003c/p\u003e\n\u003cp\u003eThese checks evaluate whether the site serves documentation in markdown format,\nwhich agents consume far more effectively than HTML.\u003c/p\u003e\n\u003ch3 id=\"markdown-url-support\" class=\"relative group\"\u003e\u003ccode\u003emarkdown-url-support\u003c/code\u003e \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#markdown-url-support\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWhat it checks\u003c/strong\u003e: Whether appending \u003ccode\u003e.md\u003c/code\u003e to documentation page URLs returns\nvalid markdown content.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhy it matters\u003c/strong\u003e: Agents work dramatically better with markdown than HTML.\nThe HTML-to-markdown conversion in web fetch pipelines is lossy and\nunpredictable. Sites that serve markdown directly bypass conversion issues\nentirely. However, agents don\u0026rsquo;t discover this pattern on their own; it needs\nto be signaled.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResult levels\u003c/strong\u003e:\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ePass\u003c/strong\u003e: \u003ccode\u003e.md\u003c/code\u003e URLs return valid markdown with 200 status.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWarn\u003c/strong\u003e: Some pages support \u003ccode\u003e.md\u003c/code\u003e but not consistently.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFail\u003c/strong\u003e: \u003ccode\u003e.md\u003c/code\u003e URLs return errors or HTML.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRecommended action\u003c/strong\u003e:\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWarn\u003c/strong\u003e: Ensure all documentation pages serve markdown when \u003ccode\u003e.md\u003c/code\u003e is\nappended to the URL, not just some.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFail\u003c/strong\u003e: Configure your docs platform to serve \u003ccode\u003e.md\u003c/code\u003e variants for all\ndocumentation pages.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAutomation\u003c/strong\u003e: Full. Test against a sample of page URLs (from \u003ccode\u003ellms.txt\u003c/code\u003e,\nsitemap, or user-provided list).\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"content-negotiation\" class=\"relative group\"\u003e\u003ccode\u003econtent-negotiation\u003c/code\u003e \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#content-negotiation\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWhat it checks\u003c/strong\u003e: Whether the server responds to \u003ccode\u003eAccept: text/markdown\u003c/code\u003e\nwith markdown content and an appropriate \u003ccode\u003eContent-Type\u003c/code\u003e header.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhy it matters\u003c/strong\u003e: Some agents (Claude Code, Cursor, OpenCode) send\n\u003ccode\u003eAccept: text/markdown\u003c/code\u003e as their preferred content type. If the server\nhonors this, the agent gets clean markdown without needing to know about\n\u003ccode\u003e.md\u003c/code\u003e URL patterns. Most agents don\u0026rsquo;t request markdown, but the ones that\ndo should get it.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResult levels\u003c/strong\u003e:\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ePass\u003c/strong\u003e: Server returns markdown content with \u003ccode\u003eContent-Type: text/markdown\u003c/code\u003e\nwhen requested.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWarn\u003c/strong\u003e: Server returns markdown content but with incorrect \u003ccode\u003eContent-Type\u003c/code\u003e.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFail\u003c/strong\u003e: Server ignores the \u003ccode\u003eAccept\u003c/code\u003e header and returns HTML regardless.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRecommended action\u003c/strong\u003e:\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWarn\u003c/strong\u003e: Set the response \u003ccode\u003eContent-Type\u003c/code\u003e to \u003ccode\u003etext/markdown\u003c/code\u003e when serving\nmarkdown content. The correct header enables optimizations in some agent\npipelines.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFail\u003c/strong\u003e: Configure your server to honor \u003ccode\u003eAccept: text/markdown\u003c/code\u003e requests\nand return markdown content. Some agents (Claude Code, Cursor, OpenCode)\nrequest markdown this way.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAutomation\u003c/strong\u003e: Full.\u003c/li\u003e\n\u003c/ul\u003e","title":"Category 2: Markdown Availability"},{"content":"Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.\nThese checks evaluate whether page content fits within the transfer and processing limits of agent web fetch pipelines. Truncation is silent: the agent doesn\u0026rsquo;t know it\u0026rsquo;s working with partial data.\nHow Agents Get Content #Not all agents see the same thing. The format an agent receives depends on the request it makes and the server\u0026rsquo;s response:\nAgents that request markdown (Claude Code, Cursor, OpenCode send Accept: text/markdown). If the server honors this and returns markdown, the agent gets clean content. If the server also returns Content-Type: text/markdown and the content is under 100K characters, Claude Code bypasses its summarization model entirely, delivering the content directly to the agent. This is the best-case path.\nAgents that request HTML (most agents, including Gemini, Copilot, and others, send Accept: text/html or */*). These agents receive the full HTML response. Some pipelines convert HTML to markdown before truncation (Claude Code uses Turndown); others may truncate raw HTML or use their own processing. The HTML path is where boilerplate CSS/JS causes the most damage.\nAgents that use .md URL variants. If an agent knows to append .md to a URL (because llms.txt told it, or a directive on the page, or persistent context), it gets markdown directly regardless of Accept headers.\nBecause different agents hit different paths, this spec defines size checks for both the markdown response (if available) and the HTML response. A site that\u0026rsquo;s only optimized for the markdown path is leaving most agents behind.\nPipelines also differ in when they cut oversized content, and the order determines which measurement predicts the agent\u0026rsquo;s experience:\nConvert, then truncate. Scripts and styles are stripped, HTML is converted to markdown, and the size limit applies to the converted output. Post-conversion size (page-size-html) predicts these pipelines. Truncate, then convert (or ingest raw HTML directly). The size limit applies to the bytes as served, so inline scripts and serialized data consume the budget before any content does. Served size (page-size-transfer) predicts these pipelines. Capped fetch. Independent of processing order, some tools cap the response bytes they will read, so a page can fail at the transfer layer before any conversion happens. Served size predicts this too. A page can score well on one measurement while failing agents on the other, which is why the spec measures both.\nFor empirical observations of how specific platforms (Claude, Cursor, Copilot, Gemini, Windsurf Cascade, and others) handle retrieval, truncation, and summarization in practice, see Agent platform comparisons.\nrendering-strategy # What it checks: Whether the HTTP response contains the page\u0026rsquo;s actual content, or whether content requires JavaScript execution to render (client-side rendering / SPA).\nWhy it matters: Most coding agents fetch pages using HTTP libraries that do not execute JavaScript. GitHub Copilot is the only major agent observed to use headless browser rendering. When a site relies on client-side rendering, agents see an empty shell containing framework boilerplate, inline CSS, and navigation chrome, but none of the documentation content.\nThis is not a truncation problem. It is a zero-content problem. The page returns HTTP 200, so the agent doesn\u0026rsquo;t know anything is wrong. It attempts to extract information from whatever text is in the shell (typically nav links and footer text) and produces nonsensical results, or falls back on training data that may be outdated.\nThe rendering strategy is a property of the framework configuration, not the framework itself. The same framework can produce either server-rendered or client-rendered output. Sites built with Next.js, Gatsby, and Nuxt appear on both sides: react.dev (Next.js) and docs.github.com (Next.js) are fully agent-accessible, while other sites using the same frameworks deliver empty shells. Text-to-HTML ratio alone is not a reliable signal; GitHub docs and Stripe docs have low ratios due to heavy bundled assets but contain real page content. The distinction is whether page-specific content is present in the response.\nA subtler variant exists where a page is statically generated but a specific component defers content rendering to JavaScript based on user selections (e.g., query parameters choosing a language or deployment type). The static HTML contains the page structure (title, navigation, selector UI) but none of the substantive content. From an agent\u0026rsquo;s perspective, the effect is the same as a full SPA shell.\nResult levels:\nPass: HTTP response contains substantive page content. Detected by the presence of multiple page-specific headings, paragraphs with prose content, or other content elements beyond navigation chrome. Warn: HTTP response contains some content but appears sparse relative to the page\u0026rsquo;s apparent scope. This covers client-side content population (statically generated pages where a component defers content to JavaScript), partial hydration or lazy loading, and legitimately minimal pages. Fail: HTTP response is an SPA shell. Detected by the combination of known framework markers (e.g., id=\u0026quot;___gatsby\u0026quot;, id=\u0026quot;__next\u0026quot;, id=\u0026quot;__nuxt\u0026quot;, id=\u0026quot;root\u0026quot;), minimal visible text content, and absence of page-specific content elements. Recommended action:\nWarn: Verify that key content is present in the server-rendered HTML response. Pages with sparse content may rely on client-side JavaScript to populate. Fail: Enable server-side rendering or pre-rendering for documentation pages. If only specific page templates use client-side content loading, target those templates rather than rebuilding the entire site. Automation: Heuristic. Combine framework marker detection with content signal analysis (headings, paragraphs, code blocks after stripping \u0026lt;script\u0026gt;, \u0026lt;style\u0026gt;, and \u0026lt;noscript\u0026gt; elements). Framework markers alone are not conclusive since SSR sites share the same markers.\nNotes: If this check fails, size-related checks (page-size-html, content-start-position) still run but their results should be interpreted with caution, since they are measuring a shell rather than actual content. This is a recommendation for report consumers and implementations presenting results; it does not require downstream checks to declare a dependency on rendering-strategy or alter their own pass/warn/fail logic. If the site passes markdown-url-support or content-negotiation, that provides partial mitigation: agents that request markdown may still get content even when the HTML path is broken.\npage-size-markdown # What it checks: The character count of the page when served as markdown, via either the .md URL variant or content negotiation with Accept: text/markdown. Only runs if the site serves markdown (as detected by Category 2 checks). Why it matters: This is the best-case scenario for agent consumption. Markdown is what agents actually want, and it\u0026rsquo;s the format where page size most directly corresponds to what the model sees. If the markdown version fits within truncation limits, agents that can request it will get the full content. Result levels: Pass: Under 50,000 characters (fits comfortably within all known limits, including Claude Code\u0026rsquo;s direct-delivery threshold for trusted sites). Warn: Between 50,000 and 100,000 characters (fits within Claude Code\u0026rsquo;s truncation limit but may exceed others; also exceeds the direct-delivery threshold, meaning a summarization model may process it). Fail: Over 100,000 characters (will be truncated by Claude Code and likely all other platforms). Recommended action: Warn: Consider splitting large pages. Pages in this range may be truncated on some platforms or routed through a summarization model. Fail: Break oversized pages into smaller ones, or restructure serialized tabbed content that inflates page size. Automation: Full. Notes: If the site doesn\u0026rsquo;t serve markdown at all, this check is skipped and page-size-html becomes the primary size check. The report should note that the site relies entirely on the HTML path. page-size-html # What it checks: The character count of the HTML response, and the character count after converting HTML to markdown (simulating what an agent\u0026rsquo;s processing pipeline produces). Reports both numbers. Why it matters: Most agents receive HTML, not markdown. The raw HTML size determines whether the page even fits in the fetch buffer (Claude Code caps at ~10MB). The post-conversion size is closer to what the agent actually processes, but conversion pipelines vary across agents and are lossy and unpredictable. Navigation boilerplate, serialized tabbed content, and deeply nested page structure can all inflate the converted output well beyond the documentation content itself. Both raw and post-conversion sizes matter. Result levels (based on post-conversion size, since that\u0026rsquo;s what the model receives): Pass: Converted content under 50,000 characters. Warn: Converted content between 50,000 and 100,000 characters. Fail: Converted content over 100,000 characters. Recommended action: Warn: Review pages for reducible boilerplate (navigation, serialized tabbed content). Consider providing markdown versions as a smaller alternative path for agents. Fail: Break large pages into smaller units, reduce navigation boilerplate, or provide markdown versions that bypass the HTML conversion overhead. Markdown availability helps agents that request it but does not reduce the HTML page size itself; fixing the HTML remains important. Automation: Full. Convert HTML to markdown using a pipeline that approximates what agents see after their own processing. Agent pipelines vary (some strip \u0026lt;script\u0026gt;/\u0026lt;style\u0026gt; elements before conversion, others don\u0026rsquo;t; some use Turndown, others use different converters or visit pages in-browser). Implementations should document their conversion approach. Report details: Show both the raw HTML size and the post-conversion size. A large gap between the two indicates heavy boilerplate. Report the conversion ratio (e.g., \u0026ldquo;505KB HTML -\u0026gt; 12KB markdown (98% boilerplate)\u0026rdquo;) as a useful signal for site owners. page-size-transfer # What it checks: The served byte size of the HTML document response: the response body after transfer decoding (decompression), which is what an agent\u0026rsquo;s HTTP client hands to its processing pipeline. Subresources (linked CSS, JavaScript, images) are not counted, because agents generally don\u0026rsquo;t fetch them. Inline scripts, styles, and serialized data payloads embedded in the document are counted, because agents can\u0026rsquo;t avoid receiving them.\nWhy it matters: page-size-html measures what survives HTML-to-markdown conversion, which models pipelines that strip scripts and convert before truncating. Served size measures what every agent pays before any processing happens, and it fails differently:\nTruncate-first and raw-ingestion pipelines consume script payload as content. For them, serialization overhead isn\u0026rsquo;t invisible; it is the page. Fetch caps apply to response bytes, not converted output. A page can convert to a few kilobytes of clean markdown and still exceed the byte budget of the tool fetching it (Claude Code\u0026rsquo;s fetch buffer caps at ~10MB). Bandwidth and latency costs apply to every fetch regardless of pipeline, and multi-page reading sessions multiply them. Modern server-rendering frameworks can make the gap between served bytes and content arbitrarily large. In measurements of production documentation sites on one hosted platform, pages shipped 75-84% of their bytes as serialized framework payloads inside inline script tags: component trees, resolved metadata, and a complete duplicate of the page\u0026rsquo;s markdown source. Served-bytes-to-content ratios ran from 40:1 to 200:1. Those pages score well on page-size-html because conversion strips the payload, while every agent fetching them transfers half a megabyte to several megabytes per page.\nResult levels:\nPass: Served size under 1MB. Warn: Served size between 1MB and 10MB. No documented cap is exceeded, but truncate-first and raw-ingestion pipelines are consuming mostly non-content bytes, and multi-page sessions pay a real bandwidth and latency cost. Fail: Served size over 10MB. This exceeds the only well-documented transfer cap (Claude Code\u0026rsquo;s fetch buffer); content beyond the cap is unreachable regardless of how the pipeline processes it. Recommended action:\nWarn: Identify what the non-content bytes are; in practice they are usually inline serialization (framework hydration payloads, embedded duplicate page source, resolved data objects) visible in the page source. Avoid shipping the same content twice in different formats, load large data payloads on demand, and confirm markdown variants are available and discoverable so agents have a cheaper path. Fail: Same actions, urgently. At this size, at least one major platform cuts the page off at the transfer layer. Markdown availability gives agents that discover it an escape hatch but does not reduce what HTML-path agents transfer; fixing the served payload remains important. Automation: Full. Fetch with an Accept-Encoding typical of agent HTTP clients, decode the response, and measure the decoded body. Implementations may also report the on-the-wire (compressed) size where available; serialized payloads compress well, so wire size understates the processing burden.\nReport details: Show served bytes alongside the post-conversion content size from page-size-html, and report the ratio between them (e.g., \u0026ldquo;3.4MB served -\u0026gt; 29KB content (~120:1)\u0026rdquo;). A high ratio on a large page is an architecture signature (hydration payloads, embedded duplicate content) rather than a content problem; it tells the site owner the fix lives in framework configuration, not in the docs themselves.\nNotes: This check complements rendering-strategy, which catches pages that ship too little server-rendered content. This check catches the opposite failure: pages that render content fine but ship many times its weight in serialization overhead. Byte-level caps are less documented than character-level truncation limits, so the default thresholds here are conservative and should be configurable (see Appendix A).\ncontent-start-position # What it checks: How far into the post-conversion content (by character count and as a percentage) the actual documentation content begins. Why it matters: After HTML-to-markdown conversion, boilerplate often survives. Navigation menus, breadcrumbs, sidebars, and footer content all convert to text that precedes or surrounds the actual documentation. Depending on the agent\u0026rsquo;s conversion pipeline, inline CSS and JavaScript may also survive as raw text. If this boilerplate consumes most of the truncation budget, the agent never sees the documentation content. In one observed case, actual content didn\u0026rsquo;t start until 87% of the way through the output (441K characters of CSS before the first paragraph). Result levels: Pass: Content starts within the first 10% of the post-conversion output. Warn: Content starts between 10% and 50%. Fail: Content starts after 50%. Recommended action: Reduce navigation, breadcrumb, and sidebar markup that precedes the content area. Agents may never see the documentation content if boilerplate consumes most of the truncation budget. Automation: Heuristic. Use the same conversion pipeline as page-size-html, then detect the first meaningful content element (heading, paragraph with prose) past any boilerplate (navigation text, breadcrumbs, sidebar content, inline CSS/JS that survived conversion). Notes: This check only applies to the HTML path. Markdown served directly by the site should not have boilerplate preamble; if it does, that\u0026rsquo;s a separate issue worth flagging but not something this check targets. single-fetch-completeness # What it checks: Whether a markdown response delivers its complete content in one fetch, and when it doesn\u0026rsquo;t, whether the continuation is machine-followable: declared where agents will see it, linked with an absolute URL, and actually working.\nWhy it matters: Pagination is application-level truncation, and it is quieter than the platform truncation this category otherwise measures. A paginated markdown response looks complete: it is well-formed, ends cleanly, and returns 200. The signals that it is partial are easy for agent pipelines to lose:\nSummarization pipelines process fetched content through a smaller model before the orchestrating agent sees it. A pagination note may or may not survive summarization, and the summarizer cannot perform a follow-up fetch itself; the orchestrator would have to notice the note and choose to fetch again. Trailing pagination notes sit at the end of the content, which is the first region lost to platform truncation. A truncated response loses the only indication that it was also paginated. RAG pipelines chunk fetched content for retrieval. A pagination marker lands in one chunk, unrelated to the content it describes, and effectively disappears. This failure was observed in production on a model catalog\u0026rsquo;s markdown variant: 100 of 102 entries shown, a pagination note at the bottom of the file, a continuation URL that was root-relative rather than absolute, and, when fetched, a continuation response that returned an empty body. An agent fetching that page gets 98% of the catalog and no working way to learn what\u0026rsquo;s missing.\nNotably, pagination in a markdown variant is often inherited from the HTML UI rather than needed by the markdown itself. The catalog above paginates at 100 entries per page, but the complete set serializes to roughly 32,000 characters, comfortably under this spec\u0026rsquo;s 50,000-character pass threshold. The markdown variant imported an interaction pattern from a surface that has interaction; markdown doesn\u0026rsquo;t.\nResult levels:\nPass: The response is complete in one fetch (no pagination signals detected), or pagination exists and the continuation is declared at the top of the content with an absolute URL that resolves to the next segment. Warn: Pagination exists and the continuation works, but is fragile: declared only at the bottom of the content, linked with a relative URL, or discoverable only from response headers. Fail: The content is partial and the continuation is missing, relative and unresolvable, or broken (non-success status, empty body, or a soft 404). Recommended action:\nWarn: Move the continuation declaration to the top of the content (before anything truncation could remove) and make continuation links absolute. Fail: First ask whether the markdown variant needs pagination at all: complete content that fits within this spec\u0026rsquo;s size thresholds should be served in one response, even when the HTML UI paginates. If pagination is genuinely necessary, declare it at the top with absolute links, and verify the continuation URLs actually serve content. Automation: Heuristic. Detect pagination signals in markdown responses: \u0026ldquo;N of M\u0026rdquo; phrasing, links or instructions containing pagination query parameters (?page=, ?offset=), \u0026ldquo;next page\u0026rdquo; link text, and Link: rel=\u0026quot;next\u0026quot; response headers. When signals are found, fetch the continuation and verify it returns substantive content of the expected representation. Absence of signals is treated as complete; a page that omits content with no marker at all is not detectable by this check (see markdown-content-parity for the cross-representation comparison that can catch it).\nNotes: Only applies to markdown responses (.md variants, content negotiation, and llms.txt-linked markdown). HTML pagination is an interaction pattern agents share with human readers and is out of scope here.\nRelationship to the spec\u0026rsquo;s splitting recommendations. This check targets windowing rather than the number of fetches. Other parts of this spec recommend more fetches: progressive disclosure splits an oversized llms.txt into section files, and the page-size checks recommend breaking large pages up. The difference is what each fetch returns. Splitting creates self-contained units, each with its own topic and URL, reached by navigation: an agent fetches an index, chooses the relevant unit, and gets content that is complete as the thing it claims to be. Pagination slices one logical unit into arbitrary windows: no window is complete for any question, every window is required to have the unit at all, and the seams are easy to lose. Multi-fetch by choice is navigation; multi-fetch by obligation is truncation with extra steps. The principle behind both: split by meaning, don\u0026rsquo;t window by size. A response should either be complete as what it claims to be, or say so plainly where agents will see it.\n","date":null,"permalink":"https://agentdocsspec.com/spec/web/page-size/","section":"Specifications","summary":"\u003cp\u003ePart of the \u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e. The \u003ca href=\"https://agentdocsspec.com/spec/web/#checks-summary\" target=\"_blank\" rel=\"noreferrer\"\u003eChecks Summary\u003c/a\u003e lists all checks with links to their definitions.\u003c/p\u003e\n\u003cp\u003eThese checks evaluate whether page content fits within the transfer and\nprocessing limits of agent web fetch pipelines. Truncation is silent: the\nagent doesn\u0026rsquo;t know it\u0026rsquo;s working with partial data.\u003c/p\u003e\n\u003ch3 id=\"how-agents-get-content\" class=\"relative group\"\u003eHow Agents Get Content \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#how-agents-get-content\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cp\u003eNot all agents see the same thing. The format an agent receives depends on the\nrequest it makes and the server\u0026rsquo;s response:\u003c/p\u003e","title":"Category 3: Page Size and Truncation Risk"},{"content":"Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.\nThese checks evaluate whether page content is structured in ways that agents can effectively consume. These are harder to fully automate and rely more on heuristics.\ntabbed-content-serialization # What it checks: Whether pages use tabbed, accordion, or dropdown UI patterns that serialize into long sequential content in the source, and if so, how large the serialized output is. Why it matters: Tabbed content is great for humans but can be catastrophic for agents. A tutorial with 11 language variants serializes into a single massive document where an agent might see only the first 1-3 variants. Source order determines what the agent sees; everything past the truncation point is invisible. Asking for a specific variant (e.g., Python) does not help if that variant is beyond the truncation point. Result levels: Pass: No tabbed content, or tabbed content that serializes to under 50,000 characters total. Warn: Tabbed content serializes to 50,000-100,000 characters. Fail: Tabbed content serializes to over 100,000 characters. Recommended action: Break tab variants into separate pages, or provide a mechanism for agents to request specific variants. Agents see only the first few variants; content in later tabs is truncated. Automation: Heuristic. Detect common tab/accordion component patterns (e.g., \u0026lt;Tab\u0026gt;, \u0026lt;Tabs\u0026gt;, role=\u0026ldquo;tabpanel\u0026rdquo;, common CSS class patterns) and estimate serialized size. section-header-quality # What it checks: Whether section headers contain enough context to be meaningful without the surrounding UI. Specifically, when tabbed content is serialized, do headers distinguish which variant (language, platform, deployment type) a section belongs to? Why it matters: When an agent sees serialized tabbed content, descriptive headers are the only way it can tell which section applies to which context. Generic headers like \u0026ldquo;Step 1\u0026rdquo; repeated across all variants are indistinguishable. Headers like \u0026ldquo;Step 1 (Python/PyMongo)\u0026rdquo; preserve the filtering context that the UI provided to human readers. Result levels (evaluated both within individual tab groups and across tab groups on the same page; the overall result is the worst of both): Pass: \u0026lt;=25% of headers within tabbed sections are generic (repeated across variants without distinguishing context). Warn: 25-50% of headers are generic across variants. Fail: \u0026gt;50% of headers are generic, or identical header sets are repeated across separate tab groups on the same page with no variant context. These thresholds are defaults; implementations should allow them to be configured. Recommended action: Add variant context to headers (e.g., \u0026ldquo;Step 1 (Python)\u0026rdquo; instead of \u0026ldquo;Step 1\u0026rdquo;) so agents can distinguish which section belongs to which variant when content is serialized. Automation: Heuristic. Requires detecting tabbed sections and analyzing header patterns within them. markdown-code-fence-validity # What it checks: Whether markdown content contains unclosed or improperly nested code fences (``` or ~~~ blocks without a matching closing delimiter). Why it matters: An unclosed code fence causes everything after it to be interpreted as code rather than prose. The agent sees documentation text, API descriptions, and instructions as if they were inside a code block, which fundamentally changes how it processes the content. A model treats code blocks as literal content to reproduce or analyze, not as natural language instructions to follow. If an unclosed fence appears early in a page, the agent effectively loses the rest of the document\u0026rsquo;s meaning. This applies to any markdown the site serves directly: pages via .md URLs or content negotiation, and llms.txt files themselves. Result levels: Pass: All code fences in the markdown content are properly opened and closed. Fail: One or more unclosed code fences detected. Recommended action: Ensure every opening ``` or ~~~ has a matching closing delimiter. Everything after an unclosed fence is interpreted as code, causing agents to misread documentation as literal content. Notes on delimiter matching: Per the CommonMark spec, a backtick fence (```) can only be closed by another backtick fence of equal or greater length, and likewise for tilde fences (~~~). Opening with ``` and attempting to close with ~~~ leaves the backtick fence unclosed. There is no intermediate \u0026ldquo;mismatched but balanced\u0026rdquo; state; mismatched delimiters produce unclosed fences and should be reported as failures. Automation: Full. Parse the markdown for fence delimiters (``` and ~~~, with optional info strings) and verify each opening delimiter has a matching close. Run against markdown served via .md URLs, content negotiation responses, and llms.txt files. Notes: This check applies to markdown the site authors and serves directly. Code fences broken by an HTML-to-markdown conversion pipeline are outside the site owner\u0026rsquo;s control, though implementations may optionally flag them as informational findings. markdown-link-portability # What it checks: Whether links in served markdown are absolute URLs, and whether a sample of them resolves to the representation they promise (a .md link returns markdown content, not an HTML error page).\nWhy it matters: Relative URL resolution is well-defined (RFC 3986), but it requires knowing the base URL, and agent pipelines routinely lose it. A browser always carries the base; markdown fetched by an agent passes through summarization models, gets chunked for RAG, or gets pasted into a context where the source URL is gone. Once the base is lost, a root-relative link is unreconstructable and a path-relative link is meaningless. This spec already recommends absolute URLs between llms.txt levels for the same reason; served markdown deserves the same rule.\nLink verification must go beyond status codes. In one observed production case, a catalog\u0026rsquo;s markdown variant emitted over 100 well-formatted links that all pointed into a wrong internal path prefix, apparently a build-time substitution error. Every link returned 200 with a body. The body was an HTML SPA shell whose only acknowledgment of failure was a serialized framework error digest inside script payload: a soft 404 served as HTML at a .md URL. A checker (or agent) that tested status codes alone would conclude the links worked; checking the Content-Type header alone would have caught it.\nResult levels:\nPass: Links are absolute URLs, and sampled links resolve to the expected representation. Warn: Links are root-relative (resolvable while the base URL is known, fragile once it isn\u0026rsquo;t), or sampled links resolve with minor mismatches (e.g., a .md link that redirects to an HTML page with the right content). Fail: Links are path-relative, or sampled links are broken: hard 404s, soft 404s, or a content type that contradicts the link (.md links returning HTML shells). Recommended action:\nWarn: Emit absolute URLs when generating markdown variants; the site\u0026rsquo;s canonical host is known at build time. Fail: Fix the link generation first, then make the links absolute. Verify generated links in CI by fetching a sample and checking both status and content type; a link set that is generated is a link set that can break wholesale. Automation: Full. Parse links from served markdown, classify as absolute, root-relative, or path-relative, resolve a sample against the fetch URL, and verify status, Content-Type, and soft-404 heuristics (reusing the detection from http-status-codes).\nNotes: Applies to markdown served via .md URLs, content negotiation, and pages discovered through llms.txt links. Links in llms.txt itself are covered by llms-txt-links-resolve, which applies the same representation verification. Same-document fragment links (#anchor with no path) are exempt from the absolute-URL requirement: they resolve within the content the agent already holds, and rewriting them to absolute URLs adds nothing.\nWhy this check exempts the HTML path. Relative links in HTML are correct web practice (they are what makes staging domains, mirrors, and CDN setups work), and the HTML path does not need the site\u0026rsquo;s help: a pipeline converting HTML to markdown still holds the fetch URL at conversion time, so resolving relative links is the pipeline\u0026rsquo;s job, with full information. The base URL is only lost downstream of conversion. Served markdown is different on both ends: the generator knows the canonical host, so absolute links are free to produce, and on the best-case consumption path (direct delivery of text/markdown under the summarization threshold) the site\u0026rsquo;s bytes reach the model verbatim, with no conversion step where anything could be resolved. Tool builders implementing fetch pipelines should resolve relative links during HTML-to-markdown conversion, and likewise when ingesting raw HTML; converted content with relative links has all the same failure modes as served markdown once it leaves the converter.\nembedded-data-serialization # What it checks: Whether machine-generated bulk data (large uniform tables, inline JSON or data blobs, base64 payloads) dominates a page\u0026rsquo;s converted content, and attributes the page\u0026rsquo;s size to the specific elements responsible.\nWhy it matters: Dynamic widgets (compatibility matrices, model catalogs, spec browsers, pricing tables) flatten into static content when a page is rendered for agents. The result can be a page whose size wildly exceeds what its author believes they wrote: the author sees a few paragraphs and a widget; the built page carries hundreds of serialized rows under them. The size checks in Category 3 catch the symptom but don\u0026rsquo;t explain it, and without attribution the person who can fix the page has no idea what to fix, or that anything is wrong at all.\nIn one measured production case, a reference page served 302KB of HTML of which 64% was table markup, including a single generated table of 218 rows. The page converts to roughly 83,000 characters (this spec\u0026rsquo;s warn band), while its non-table prose totals about 17,000 characters. The page\u0026rsquo;s truncation risk is entirely a property of its generated tables, which is invisible in an aggregate size number.\nResult levels:\nPass: No bulk-data elements detected, or bulk elements are present but the page passes the Category 3 size checks regardless. Warn: Bulk-data elements are the dominant contributor (for example, over half of converted content) to a page that lands in the size checks\u0026rsquo; warn band. Fail: Bulk-data elements are the dominant contributor to a page that exceeds the size checks\u0026rsquo; fail threshold. Content after the bulk element is beyond the truncation point for most platforms. Recommended action: Bulk data is often legitimate content (a support matrix is the point of a support-matrix page), so the goal is structure, not removal. Split large generated tables across per-section pages (as self-contained units reached from an index, each complete for its scope; paginating one table into windows trades this problem for the one single-fetch-completeness describes), provide filtered or queryable views, load embedded data blobs on demand, and place prose before bulk elements so truncation removes data rows rather than explanation. Report the attribution to content authors: a page that an author experiences as two paragraphs should not ship as a hundred kilobytes without the author knowing.\nAutomation: Heuristic. After HTML-to-markdown conversion (same pipeline as page-size-html), detect bulk elements: tables above a row threshold with uniform row structure, fenced or inline JSON blobs above a size threshold, and base64 runs. Report each element\u0026rsquo;s share of the converted content. Thresholds for \u0026ldquo;bulk\u0026rdquo; need calibration against real pages and should be configurable. This check sees only bulk that survives conversion into content (tables, code fences, prose-embedded data); serialized payloads inside \u0026lt;script\u0026gt; tags are stripped by conversion and are page-size-transfer\u0026rsquo;s domain. The two checks divide the data-dump problem by pipeline: script payloads burden every fetch, content-embedded bulk burdens what the model reads.\nNotes: This is the data-widget sibling of tabbed-content-serialization, which covers the same flattening failure for tab and accordion UI. Together with content-start-position (where content sits relative to boilerplate), these checks explain why a page fails the Category 3 size checks, not just that it does.\n","date":null,"permalink":"https://agentdocsspec.com/spec/web/content-structure/","section":"Specifications","summary":"\u003cp\u003ePart of the \u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e. The \u003ca href=\"https://agentdocsspec.com/spec/web/#checks-summary\" target=\"_blank\" rel=\"noreferrer\"\u003eChecks Summary\u003c/a\u003e lists all checks with links to their definitions.\u003c/p\u003e\n\u003cp\u003eThese checks evaluate whether page content is structured in ways that agents can\neffectively consume. These are harder to fully automate and rely more on\nheuristics.\u003c/p\u003e\n\u003ch3 id=\"tabbed-content-serialization\" class=\"relative group\"\u003e\u003ccode\u003etabbed-content-serialization\u003c/code\u003e \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#tabbed-content-serialization\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWhat it checks\u003c/strong\u003e: Whether pages use tabbed, accordion, or dropdown UI\npatterns that serialize into long sequential content in the source, and if\nso, how large the serialized output is.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhy it matters\u003c/strong\u003e: Tabbed content is great for humans but can be catastrophic\nfor agents. A tutorial with 11 language variants serializes into a single\nmassive document where an agent might see only the first 1-3 variants. Source\norder determines what the agent sees; everything past the truncation point is\ninvisible. Asking for a specific variant (e.g., Python) does not help if that\nvariant is beyond the truncation point.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResult levels\u003c/strong\u003e:\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ePass\u003c/strong\u003e: No tabbed content, or tabbed content that serializes to under\n50,000 characters total.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWarn\u003c/strong\u003e: Tabbed content serializes to 50,000-100,000 characters.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFail\u003c/strong\u003e: Tabbed content serializes to over 100,000 characters.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRecommended action\u003c/strong\u003e: Break tab variants into separate pages, or provide\na mechanism for agents to request specific variants. Agents see only the\nfirst few variants; content in later tabs is truncated.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAutomation\u003c/strong\u003e: Heuristic. Detect common tab/accordion component patterns\n(e.g., \u003ccode\u003e\u0026lt;Tab\u0026gt;\u003c/code\u003e, \u003ccode\u003e\u0026lt;Tabs\u0026gt;\u003c/code\u003e, role=\u0026ldquo;tabpanel\u0026rdquo;, common CSS class patterns) and\nestimate serialized size.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"section-header-quality\" class=\"relative group\"\u003e\u003ccode\u003esection-header-quality\u003c/code\u003e \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#section-header-quality\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWhat it checks\u003c/strong\u003e: Whether section headers contain enough context to be\nmeaningful without the surrounding UI. Specifically, when tabbed content is\nserialized, do headers distinguish which variant (language, platform,\ndeployment type) a section belongs to?\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhy it matters\u003c/strong\u003e: When an agent sees serialized tabbed content, descriptive\nheaders are the only way it can tell which section applies to which context.\nGeneric headers like \u0026ldquo;Step 1\u0026rdquo; repeated across all variants are\nindistinguishable. Headers like \u0026ldquo;Step 1 (Python/PyMongo)\u0026rdquo; preserve the\nfiltering context that the UI provided to human readers.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResult levels\u003c/strong\u003e (evaluated both within individual tab groups and across\ntab groups on the same page; the overall result is the worst of both):\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ePass\u003c/strong\u003e: \u0026lt;=25% of headers within tabbed sections are generic (repeated\nacross variants without distinguishing context).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWarn\u003c/strong\u003e: 25-50% of headers are generic across variants.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFail\u003c/strong\u003e: \u0026gt;50% of headers are generic, or identical header sets are\nrepeated across separate tab groups on the same page with no variant\ncontext.\nThese thresholds are defaults; implementations should allow them to be\nconfigured.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRecommended action\u003c/strong\u003e: Add variant context to headers (e.g., \u0026ldquo;Step 1\n(Python)\u0026rdquo; instead of \u0026ldquo;Step 1\u0026rdquo;) so agents can distinguish which section\nbelongs to which variant when content is serialized.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAutomation\u003c/strong\u003e: Heuristic. Requires detecting tabbed sections and analyzing\nheader patterns within them.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"markdown-code-fence-validity\" class=\"relative group\"\u003e\u003ccode\u003emarkdown-code-fence-validity\u003c/code\u003e \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#markdown-code-fence-validity\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWhat it checks\u003c/strong\u003e: Whether markdown content contains unclosed or improperly\nnested code fences (\u003ccode\u003e```\u003c/code\u003e or \u003ccode\u003e~~~\u003c/code\u003e blocks without a matching closing\ndelimiter).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhy it matters\u003c/strong\u003e: An unclosed code fence causes everything after it to be\ninterpreted as code rather than prose. The agent sees documentation text,\nAPI descriptions, and instructions as if they were inside a code block,\nwhich fundamentally changes how it processes the content. A model treats\ncode blocks as literal content to reproduce or analyze, not as natural\nlanguage instructions to follow. If an unclosed fence appears early in a\npage, the agent effectively loses the rest of the document\u0026rsquo;s meaning. This\napplies to any markdown the site serves directly: pages via \u003ccode\u003e.md\u003c/code\u003e URLs or\ncontent negotiation, and \u003ccode\u003ellms.txt\u003c/code\u003e files themselves.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResult levels\u003c/strong\u003e:\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ePass\u003c/strong\u003e: All code fences in the markdown content are properly opened and\nclosed.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFail\u003c/strong\u003e: One or more unclosed code fences detected.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRecommended action\u003c/strong\u003e: Ensure every opening \u003ccode\u003e```\u003c/code\u003e or \u003ccode\u003e~~~\u003c/code\u003e has a\nmatching closing delimiter. Everything after an unclosed fence is\ninterpreted as code, causing agents to misread documentation as literal\ncontent.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNotes on delimiter matching\u003c/strong\u003e: Per the CommonMark spec, a backtick fence\n(\u003ccode\u003e```\u003c/code\u003e) can only be closed by another backtick fence of equal or greater\nlength, and likewise for tilde fences (\u003ccode\u003e~~~\u003c/code\u003e). Opening with \u003ccode\u003e```\u003c/code\u003e and\nattempting to close with \u003ccode\u003e~~~\u003c/code\u003e leaves the backtick fence unclosed. There is\nno intermediate \u0026ldquo;mismatched but balanced\u0026rdquo; state; mismatched delimiters\nproduce unclosed fences and should be reported as failures.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAutomation\u003c/strong\u003e: Full. Parse the markdown for fence delimiters (\u003ccode\u003e```\u003c/code\u003e and\n\u003ccode\u003e~~~\u003c/code\u003e, with optional info strings) and verify each opening delimiter has a\nmatching close. Run against markdown served via \u003ccode\u003e.md\u003c/code\u003e URLs, content\nnegotiation responses, and \u003ccode\u003ellms.txt\u003c/code\u003e files.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eNotes\u003c/strong\u003e: This check applies to markdown the site authors and serves\ndirectly. Code fences broken by an HTML-to-markdown conversion pipeline are\noutside the site owner\u0026rsquo;s control, though implementations may optionally flag\nthem as informational findings.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"markdown-link-portability\" class=\"relative group\"\u003e\u003ccode\u003emarkdown-link-portability\u003c/code\u003e \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#markdown-link-portability\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003e\n\u003cp\u003e\u003cstrong\u003eWhat it checks\u003c/strong\u003e: Whether links in served markdown are absolute URLs, and\nwhether a sample of them resolves to the representation they promise (a\n\u003ccode\u003e.md\u003c/code\u003e link returns markdown content, not an HTML error page).\u003c/p\u003e","title":"Category 4: Content Structure"},{"content":"Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.\nThese checks evaluate whether documentation URLs behave in ways that agents can handle, given that agents retrieve URLs from training data and have limited ability to discover moved content.\nhttp-status-codes # What it checks: Whether pages return correct HTTP status codes. In particular, whether \u0026ldquo;not found\u0026rdquo; pages return 404 (not 200 with a friendly error page). Why it matters: Soft 404s (200 status with \u0026ldquo;page not found\u0026rdquo; content) are worse than real 404s for agents. The agent sees a 200 and tries to extract information from the error page content rather than recognizing the page doesn\u0026rsquo;t exist. A clean 404 tells the agent to try a different approach. Result levels: Pass: Error pages return appropriate 4xx status codes. Fail: Error pages return 200 (soft 404). Recommended action: Configure your server to return 404 status codes for pages that don\u0026rsquo;t exist. Agents try to extract information from soft 404 page content instead of recognizing the page is missing. Automation: Full. Test known-bad URLs (e.g., append random strings to real page paths) and check status codes. redirect-behavior # What it checks: Whether redirects are same-host (transparent to agents) or cross-host (a friction point), and whether redirects use proper HTTP status codes (301/302) vs. JavaScript-based redirects. Why it matters: Same-host redirects work transparently because the HTTP client follows them automatically. Cross-host redirects are a known failure point; Claude Code, for example, doesn\u0026rsquo;t automatically follow cross-host redirects (security measure against open-redirect attacks). JavaScript redirects don\u0026rsquo;t work at all because agents don\u0026rsquo;t execute JavaScript. Result levels: Pass: All redirects are same-host HTTP redirects (301/302). Warn: Cross-host HTTP redirects are present (agents may or may not follow them depending on the platform). Fail: JavaScript-based redirects are detected. Recommended action: Warn: Where possible, use same-host redirects or update URLs to point directly to the final destination. Fail: Replace JavaScript-based redirects with HTTP 301/302 redirects. Agents don\u0026rsquo;t execute JavaScript and will not follow these redirects. Automation: Partial. HTTP redirects are detectable. JavaScript redirects require fetching the page and scanning for window.location, meta refresh, or similar patterns. ","date":null,"permalink":"https://agentdocsspec.com/spec/web/url-stability/","section":"Specifications","summary":"\u003cp\u003ePart of the \u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e. The \u003ca href=\"https://agentdocsspec.com/spec/web/#checks-summary\" target=\"_blank\" rel=\"noreferrer\"\u003eChecks Summary\u003c/a\u003e lists all checks with links to their definitions.\u003c/p\u003e\n\u003cp\u003eThese checks evaluate whether documentation URLs behave in ways that agents can\nhandle, given that agents retrieve URLs from training data and have limited\nability to discover moved content.\u003c/p\u003e\n\u003ch3 id=\"http-status-codes\" class=\"relative group\"\u003e\u003ccode\u003ehttp-status-codes\u003c/code\u003e \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#http-status-codes\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWhat it checks\u003c/strong\u003e: Whether pages return correct HTTP status codes. In\nparticular, whether \u0026ldquo;not found\u0026rdquo; pages return 404 (not 200 with a friendly\nerror page).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhy it matters\u003c/strong\u003e: Soft 404s (200 status with \u0026ldquo;page not found\u0026rdquo; content) are\nworse than real 404s for agents. The agent sees a 200 and tries to extract\ninformation from the error page content rather than recognizing the page\ndoesn\u0026rsquo;t exist. A clean 404 tells the agent to try a different approach.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResult levels\u003c/strong\u003e:\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ePass\u003c/strong\u003e: Error pages return appropriate 4xx status codes.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFail\u003c/strong\u003e: Error pages return 200 (soft 404).\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRecommended action\u003c/strong\u003e: Configure your server to return 404 status codes\nfor pages that don\u0026rsquo;t exist. Agents try to extract information from soft\n404 page content instead of recognizing the page is missing.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAutomation\u003c/strong\u003e: Full. Test known-bad URLs (e.g., append random strings to real\npage paths) and check status codes.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"redirect-behavior\" class=\"relative group\"\u003e\u003ccode\u003eredirect-behavior\u003c/code\u003e \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#redirect-behavior\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWhat it checks\u003c/strong\u003e: Whether redirects are same-host (transparent to agents) or\ncross-host (a friction point), and whether redirects use proper HTTP status\ncodes (301/302) vs. JavaScript-based redirects.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWhy it matters\u003c/strong\u003e: Same-host redirects work transparently because the HTTP\nclient follows them automatically. Cross-host redirects are a known failure\npoint; Claude Code, for example, doesn\u0026rsquo;t automatically follow cross-host\nredirects (security measure against open-redirect attacks). JavaScript\nredirects don\u0026rsquo;t work at all because agents don\u0026rsquo;t execute JavaScript.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eResult levels\u003c/strong\u003e:\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003ePass\u003c/strong\u003e: All redirects are same-host HTTP redirects (301/302).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eWarn\u003c/strong\u003e: Cross-host HTTP redirects are present (agents may or may not\nfollow them depending on the platform).\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFail\u003c/strong\u003e: JavaScript-based redirects are detected.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eRecommended action\u003c/strong\u003e:\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eWarn\u003c/strong\u003e: Where possible, use same-host redirects or update URLs to point\ndirectly to the final destination.\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eFail\u003c/strong\u003e: Replace JavaScript-based redirects with HTTP 301/302 redirects.\nAgents don\u0026rsquo;t execute JavaScript and will not follow these redirects.\u003c/li\u003e\n\u003c/ul\u003e\n\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eAutomation\u003c/strong\u003e: Partial. HTTP redirects are detectable. JavaScript redirects\nrequire fetching the page and scanning for \u003ccode\u003ewindow.location\u003c/code\u003e, \u003ccode\u003emeta refresh\u003c/code\u003e,\nor similar patterns.\u003c/li\u003e\n\u003c/ul\u003e","title":"Category 5: URL Stability and Redirects"},{"content":"Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.\nThese checks evaluate whether the site\u0026rsquo;s agent-facing resources stay accurate and up to date over time. Categories 1-5 can be evaluated as point-in-time audits; this category addresses the ongoing maintenance dimension. llms.txt files and markdown endpoints are secondary outputs that often aren\u0026rsquo;t wired into existing monitoring, so they can go stale, break, or drift from primary HTML content without anyone noticing.\nllms-txt-coverage # What it checks: How much of the site\u0026rsquo;s documentation is represented in llms.txt.\nWhy it matters: llms.txt is an agent\u0026rsquo;s primary navigational index into a documentation site. Pages missing from the index are effectively invisible to agents that rely on it for discovery. Unlike llms-txt-links-resolve (which catches broken links to pages that are listed), this check catches the opposite problem: pages that exist on the site but aren\u0026rsquo;t listed at all.\nNot every gap is a problem; many sites intentionally curate their llms.txt to include only a subset of pages. The check\u0026rsquo;s value is making the coverage level visible so site owners can confirm it reflects their intent.\nResult levels (based on coverage of sitemap doc pages, excluding non-doc pages like blog posts, pricing, and login pages):\nPass: llms.txt links cover \u0026gt;=95% of the site\u0026rsquo;s primary pages. Warn: llms.txt links cover 80-95% of primary pages (some live pages are missing). Fail: llms.txt links cover \u0026lt;80% of primary pages (missing large sections of the documentation). These thresholds are defaults that assume the site intends llms.txt to mirror the sitemap. Sites that intentionally curate their llms.txt should adjust thresholds to match their intent (see Notes below). Implementations should allow thresholds to be configured. Recommended action:\nWarn: Review missing pages. If they should be in llms.txt, add them. If they are intentionally excluded, adjust the coverage threshold or add them to an exclusion list so the check reflects your intent. Fail: If unintentional, regenerate llms.txt from your sitemap or build pipeline. If intentional, lower the threshold or set it to 0 to make the check informational. Automation: Heuristic. Compare links in llms.txt against a sitemap or crawled page list; flag pages present in the sitemap but absent from llms.txt. Implementations should support exclusion patterns that remove known-intentional gaps from the sitemap before calculating coverage.\nNotes: Not every sitemap page belongs in llms.txt. Sites intentionally exclude content for good reasons: changelog and release notes archives that would bloat the file, older product versions that aren\u0026rsquo;t relevant to current development, API reference pages that aren\u0026rsquo;t useful in markdown form, or directory pages that just link to other pages already listed. This is legitimate curation, not drift.\nThe check should accommodate three use cases through configurable thresholds and exclusion patterns:\nFull parity: The site intends llms.txt to mirror the sitemap. Default thresholds (95/80) apply; no exclusions needed. Curated: The site intentionally includes only a subset of pages. Set thresholds to 0 to make the check informational. It still reports coverage percentage and lists what\u0026rsquo;s missing, but never warns or fails. Hybrid: The site wants strict coverage but with known exclusions. Exclusion patterns remove intentional gaps from the sitemap before calculating coverage; remaining pages are held to the default thresholds. The definition of \u0026ldquo;primary pages\u0026rdquo; in the denominator requires judgment. Implementations should document how they construct the URL pool from the sitemap and what filtering they apply.\nmarkdown-content-parity # What it checks: Whether markdown versions of pages contain the same substantive content as their HTML counterparts.\nWhy it matters: When markdown is generated separately from HTML (rather than being the source that HTML is built from), the two can drift. A site might update an HTML page but forget to regenerate the markdown version, leaving agents with outdated instructions or code examples. This is particularly insidious because agents that receive the markdown version have no signal that a newer HTML version exists.\nHowever, in some cases, content divergence may be intentional. Some sites intentionally serve different content to different audiences, providing agent-optimized markdown alongside human-optimized HTML. In those cases, the divergence is deliberate. The check\u0026rsquo;s value is surfacing it so site owners can confirm it reflects their intent.\nResult levels (based on the percentage of content segments in the HTML version that are missing from the markdown version, after normalizing whitespace, case, and formatting):\nPass: \u0026lt;5% of content segments missing (or page has fewer than 10 segments, which is too small to produce meaningful parity scores). Warn: 5-20% of content segments missing (minor differences: formatting variations, navigation elements present in one but not the other). Fail: \u0026gt;=20% of content segments missing (substantive content differences: missing sections, outdated code examples, or different instructions between the two versions). These thresholds are defaults that assume the site intends markdown to mirror HTML. Sites that intentionally serve different content per audience should adjust thresholds to match their intent (see Notes below). Implementations should allow thresholds to be configured. Recommended action:\nWarn: Review pages with minor differences. If they are formatting variations that may affect agent comprehension, fix them. If they reflect intentional audience segmentation, adjust thresholds or configure the check to account for it. Fail: If unintentional, agents receiving the markdown version are getting outdated or incomplete content. Regenerate markdown from source or fix the build pipeline. If intentional, lower the threshold or set it to 0 to make the check informational. Automation: Heuristic. Fetch both versions, extract text content from HTML (strip tags), and compare key sections (headings, code blocks, paragraph content) for meaningful differences. Minor formatting differences should be ignored. If the HTML contains audience-segmentation tags (see Notes), implementations should strip tagged content before comparing so that intentionally excluded content does not count as missing.\nNotes: Sites where markdown is the source format and HTML is generated from it are less likely to have parity issues, but the check is still valuable as a safety net for build pipeline failures.\nAudience segmentation. Some documentation platforms use HTML tags to control what content appears in each version. For example, a platform might tag certain content as agent-only (included in markdown but not rendered in HTML) or human-only (rendered in HTML but excluded from markdown). Platforms like Fern and Mintlify have implemented this pattern. When the HTML contains recognized audience-segmentation tags, implementations should account for them before comparing: content explicitly tagged for one audience should not count as missing from the other.\nThe spec does not define a standard set of segmentation tags or prescribe which vendor conventions to recognize. Implementations should document which tag conventions they support, and vendors or site owners who want their conventions recognized can contribute them to implementations directly.\nAs with llms-txt-coverage, the check should accommodate sites at different points on the mirrored-to-curated spectrum:\nMirrored (default): Markdown should match HTML. Default thresholds apply. Segmented: The site uses audience-segmentation tags to control per-version content. The check strips tagged content before comparing; remaining shared content is held to the default thresholds. Curated: The site intentionally serves different content with no tag-level signal. Set thresholds to 0 to make the check informational. Dynamically generated pages. Pages built from data (catalogs, model listings, compatibility matrices) can diverge between representations without anyone deciding they should, because the HTML and markdown variants are rendered by different pipelines with different defaults. In one observed production case, a catalog\u0026rsquo;s HTML showed 98 items while its markdown variant listed 102: the HTML applied a default filter the markdown dump didn\u0026rsquo;t, and the markdown was additionally paginated. For pages with repeated structure, implementations should compare item counts between representations, and should distinguish the likely causes when counts differ: a default filter on the dynamic view (divergent by configuration), pagination on either side (divergent by windowing, see single-fetch-completeness), or staleness (one representation generated from older data). Each has a different owner and fix.\ncache-header-hygiene # What it checks: Whether llms.txt and markdown endpoints have cache headers that allow timely updates. Why it matters: Aggressive caching on agent-facing resources means that even after a site owner updates their llms.txt or markdown content, agents (and intermediary CDNs) may continue serving stale versions for hours or days. Conversely, no cache headers at all leads to ambiguous behavior where different CDN providers apply their own defaults. For resources that are relatively small and infrequently fetched, short cache lifetimes with revalidation are appropriate. Result levels: Pass: Cache headers allow timely updates (e.g., max-age under 3600, or uses must-revalidate with ETag/Last-Modified). Warn: Moderate caching (1-24 hours) that could delay updates. Fail: Aggressive caching (over 24 hours) with no revalidation mechanism, or no cache-related headers at all (ambiguous behavior). An exception: responses that lack Cache-Control and Expires but include ETag or Last-Modified should pass, since these validation headers enable conditional revalidation by browsers and CDNs even without explicit cache directives. Recommended action: Warn: Updates to llms.txt or markdown content may take hours to propagate. Consider reducing cache lifetimes for these resources. Fail: Set max-age under 3600 or add must-revalidate with ETag/Last-Modified so content updates reach agents promptly. Automation: Full. Inspect Cache-Control, Expires, ETag, and Last-Modified response headers. Ongoing Monitoring Recommendations #The three checks above can be run as one-time audits, but they\u0026rsquo;re most valuable when run on a schedule. This section offers non-normative guidance on integrating agent-facing resources into existing monitoring workflows.\nInclude llms.txt and markdown endpoints in uptime monitoring. These resources should be monitored alongside your primary documentation site. A 200 response from your docs homepage doesn\u0026rsquo;t guarantee that /llms.txt or .md URL variants are also healthy. Add them to whatever uptime tool you already use (Pingdom, Uptime Robot, Checkly, etc.) as separate check targets.\nSet up alerting for response time degradation. If your llms.txt or markdown endpoints start responding slowly, agents may time out before receiving content. This is especially relevant for dynamically generated markdown (as opposed to static files), where a backend issue could cause latency spikes that don\u0026rsquo;t affect the HTML site.\nRun coverage and parity checks on a schedule. Rather than treating llms-txt-coverage and markdown-content-parity as one-time audits, run them weekly or on every deploy. A CI check that compares llms.txt link coverage against the sitemap can catch missing pages before they reach production.\nMonitor for silent failures. A 200 response with empty content, a generic error message, or a login page is worse than a clean 404, because agents will try to extract information from the response. Check that llms.txt and markdown responses contain expected content markers (e.g., an H1, a minimum character count) rather than just checking for a 200 status code.\n","date":null,"permalink":"https://agentdocsspec.com/spec/web/observability/","section":"Specifications","summary":"\u003cp\u003ePart of the \u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e. The \u003ca href=\"https://agentdocsspec.com/spec/web/#checks-summary\" target=\"_blank\" rel=\"noreferrer\"\u003eChecks Summary\u003c/a\u003e lists all checks with links to their definitions.\u003c/p\u003e\n\u003cp\u003eThese checks evaluate whether the site\u0026rsquo;s agent-facing resources stay accurate\nand up to date over time. Categories 1-5 can be evaluated as point-in-time\naudits; this category addresses the ongoing maintenance dimension. \u003ccode\u003ellms.txt\u003c/code\u003e\nfiles and markdown endpoints are secondary outputs that often aren\u0026rsquo;t wired\ninto existing monitoring, so they can go stale, break, or drift from primary\nHTML content without anyone noticing.\u003c/p\u003e","title":"Category 6: Observability and Content Health"},{"content":"Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.\nThese checks evaluate whether documentation is accessible to agents at all: without requiring interactive authentication, and without infrastructure-level barriers aimed at automated clients. Docs behind login walls are effectively invisible to coding agents, which has significant implications as agent-assisted development becomes a standard workflow. Bot-protection systems can produce the same invisibility through a different mechanism, and often without the site owner realizing documentation is affected.\nWhy This Matters #Enterprises often gate documentation behind authentication to protect intellectual property, enforce licensing terms, or comply with access control policies. These are legitimate business reasons. However, the tradeoff is sharper than most organizations realize: authenticated docs are not just inconvenient for agents, they are completely inaccessible.\nWhen an agent encounters an auth-gated page, it sees one of these:\nA 401 or 403 response, which tells it nothing useful. A login page returned as 200, which is a soft 404 from the agent\u0026rsquo;s perspective. The agent tries to extract documentation from the login form HTML and produces nonsensical results. A redirect to an SSO provider, which is a cross-host redirect the agent cannot follow, even if it wanted to. In all three cases, the agent may take one of two actions:\nFall back on whatever it absorbed during training, which may be outdated, incomplete, or wrong. Leave your official product website and look for secondary sources to learn about your product, including blogs or articles which may inaccurate, outdated, and not reflect your official best practices. In these scenarios, the developer either gets bad guidance, or has to manually copy-paste docs into the conversation, losing the workflow benefits that agents provide. This may also be completely invisible to the developer, as an agent may \u0026ldquo;helpfully\u0026rdquo; turn to blog posts or secondary references without disclosing to the human user that it used secondary sources which should be verified.\nThe competitive dimension is real. If your product\u0026rsquo;s documentation requires a login and your competitor\u0026rsquo;s doesn\u0026rsquo;t, developers using agents will have a dramatically better experience with the competitor\u0026rsquo;s product. The agent can read the competitor\u0026rsquo;s API reference, find code examples, and verify patterns in real time. For your product, the agent is guessing.\nBot protection produces the same invisibility through a different mechanism, and usually without a deliberate decision. Auth gating is at least a policy choice about who may read the docs; bot enforcement is typically configured site-wide for security reasons, with documentation caught as collateral. It is also harder for the site owner to see. A login wall fails every request the same way, but behavioral enforcement can pass a casual spot check and then throttle, challenge, or stall the sustained multi-page sessions that real agent work produces. A site owner who verifies their docs by loading a page in a browser, or fetching a single URL with curl, will conclude everything works.\nauth-gate-detection # What it checks: Whether documentation pages require authentication to access content. Why it matters: A documentation site that returns login pages, 401/403 responses, or SSO redirects for its content pages is completely opaque to agents. This check identifies the problem so site owners can make an informed decision about the tradeoff. Result levels: Pass: Documentation pages return content (200 with substantive body) without requiring authentication. Warn: Some pages are accessible but others require authentication (partial gating). This is common for sites that gate advanced content or API references while keeping tutorials public. Fail: All or most documentation pages require authentication. Recommended action: Warn: Consider ungating reference documentation and API guides. Agents can access public pages but will fall back on training data for gated content. Fail: Agents cannot access your documentation and will rely on potentially outdated training data or secondary sources. Consider providing alternative access paths (see auth-alternative-access). Automation: Full. Fetch a sample of documentation URLs and classify responses: 200 with content (accessible), 401/403 (auth required), 200 with login form heuristics (soft auth gate), or redirect to known SSO providers (auth redirect). Login form detection uses heuristics: look for \u0026lt;input type=\u0026quot;password\u0026quot;\u0026gt;, common SSO redirect domains (okta.com, auth0.com, login.microsoftonline.com), or page titles containing \u0026ldquo;sign in\u0026rdquo; or \u0026ldquo;log in\u0026rdquo;. Notes: This check is informational for sites that intentionally gate content. It doesn\u0026rsquo;t prescribe that all docs must be public. It ensures the site owner is aware of the agent accessibility impact and can evaluate whether alternative access paths (see below) are warranted. auth-alternative-access # What it checks: Whether an auth-gated documentation site provides alternative access paths that agents can use. Why it matters: Sites that must gate their primary docs can still serve agents through secondary channels. This check looks for evidence that such channels exist, giving the site credit for providing agent access even when the main docs require a login. Result levels: Pass: At least one alternative access path is detected (see list below). Warn: The site provides partial alternative access (e.g., an llms.txt exists but only covers a subset of the gated content). Fail: No alternative access paths detected for auth-gated content. Recommended action: Warn: Expand alternative access to cover more of the gated documentation. Fail: Consider providing a public llms.txt, ungating reference docs, shipping docs with your SDK, or providing an MCP server for authenticated access. See Making Private Docs Agent-Accessible for options ordered by implementation effort. Automation: Partial. Some access paths can be detected automatically; others require manual verification. Detectable access paths: Public llms.txt: The site serves an llms.txt file that doesn\u0026rsquo;t require authentication, even if the underlying docs pages do. This gives agents at least a navigational index. Public markdown or API endpoint: Some pages or a content API respond to unauthenticated requests even when the main docs UI requires login. Bundled documentation: The product ships docs as part of its package or SDK (e.g., a docs/ directory, man pages, or built-in help subcommands). Agents can read local files without authentication. CLI-based doc access: The product provides a CLI command (e.g., yourproduct docs search \u0026quot;topic\u0026quot;) that the developer has already authenticated, making content available to agents through tool use. MCP server: The organization provides an MCP server that exposes documentation through tool calls, with authentication handled server-side. This is the most capable option for private docs because it preserves full content access while keeping credentials out of the agent context. (Detection is manual; there\u0026rsquo;s no standard way to discover whether a company offers an MCP server.) Notes: Only applies when auth-gate-detection returns warn or fail. If docs are publicly accessible, this check is skipped. bot-protection-interference # What it checks: Whether bot-protection systems (CDN bot management, WAF rules, behavioral rate enforcement) interfere with automated fetching of documentation content.\nWhy it matters: Coding agents are automated clients. Bot management tuned for scraper and attack traffic frequently cannot distinguish an agent fetching docs on a developer\u0026rsquo;s behalf from abuse, and its enforcement modes are worse for agents than a clean block because the failures are invisible:\nChallenge interstitials served as 200. A \u0026ldquo;verifying your browser\u0026rdquo; page returned with a success status is a soft 404 from the agent\u0026rsquo;s perspective; the agent extracts challenge boilerplate instead of documentation and may present it as an answer. Tarpits. The server accepts the connection and returns headers, then holds the response body open indefinitely. The agent\u0026rsquo;s fetch stalls until its own timeout with no error to reason about, and a multi-page reading session dies partway through. Volume-triggered throttling or blocking. Enforcement engages only after several requests, so the first pages of a session succeed and later ones fail. Because enforcement is typically stateful (keyed to IP or client fingerprint) and decays over time, single-page spot checks look healthy while sustained agent sessions fail. These modes are not hypothetical. In one observed production case, a CDN\u0026rsquo;s bot management responded to a sustained documentation scan by holding response bodies open indefinitely. Single-request probes of the same pages looked healthy throughout, and enforcement decayed after a cooldown period.\nResult levels:\nPass: A sustained multi-page scan completes with no evidence of interference: no challenge pages, no stalled response bodies, no volume-correlated failures. Because detection is heuristic and enforcement is stateful, pass means no interference was observed during this run, not a guarantee that bot protection will never engage. Warn: Intermittent interference. Some requests during a sustained scan are challenged, stalled, or blocked while others succeed. Fail: Sustained fetching is effectively blocked. Once enforcement triggers, most requests are challenged, stalled, or denied. Recommended action:\nWarn: Identify which bot-management layer is challenging or stalling some requests and exempt public documentation routes from behavioral enforcement. Intermittent interference means enforcement thresholds sit close to normal agent reading cadence, so small configuration changes (or ordinary traffic growth) can tip it into sustained blocking. Fail: Treat public documentation paths as automation-friendly in bot-management configuration. Exempt docs routes from behavioral enforcement, or scope enforcement to interactive product surfaces. Where limits are genuinely needed, prefer an explicit 429 with Retry-After over tarpits or silent blocks: a 429 is an error the agent can see, report, and react to, while a tarpit or challenge page fails invisibly. Never serve challenge interstitials with a 200 status. Automation: Heuristic. Interference generally cannot be probed directly without generating the sustained traffic that triggers it, so implementations should detect it as a byproduct of a normal scan: response bodies that stall past the request timeout, challenge-page heuristics in fetched content, or failure rates that climb as the scan progresses. Unlike other checks, this one has no fetch phase of its own; it is evaluated from evidence accumulated across the entire run rather than probed as a discrete step.\nA scan that follows this spec\u0026rsquo;s guidance in A Note on Responsible Use already resembles a realistic multi-page agent reading session, and that is the correct calibration, because that workload is exactly what this check predicts. If a respectful scan at agent-like cadence triggers enforcement, that is the finding, not a scan artifact. Implementations should not escalate traffic to deliberately provoke enforcement. Because enforcement is stateful and decays, results vary across runs; a clean run does not prove absence.\nNotes: When interference is detected mid-scan, results from other multi-page checks are computed on whatever sample survived. Implementations should surface a run-level warning that scores may not reflect the full site (see the Bot Protection Degrading Scan Reliability interaction effect). This check identifies the condition; it does not prescribe that sites disable bot protection. Like auth gating, this is a tradeoff the site owner should make deliberately, with awareness that coding agents are among the clients being blocked.\nResults are also vantage-point dependent. Enforcement is commonly keyed to client reputation (IP range, ASN, TLS fingerprint), so a scan run from datacenter infrastructure may trigger enforcement that residential traffic would not. This mirrors real agent traffic, which originates from the same mix of vantage points: some harnesses fetch from the developer\u0026rsquo;s own connection, while others route web fetches through vendor server infrastructure or run in cloud-hosted sessions, both of which present datacenter IPs to the site. A datacenter-origin scan is representative of that second class of agent traffic, not a false positive. Implementations should note the scan\u0026rsquo;s network context alongside results; even a coarse classification (a developer machine versus CI or cloud infrastructure) lets a reader interpret enforcement findings correctly. Reports should carry the classification rather than the scanner\u0026rsquo;s raw IP address, since reports are often shared.\nThis check is distinct from robots.txt and AI user-agent blocking, which this spec intentionally excludes (see Appendix B). Declared crawling policy is invisible to most coding agents because they don\u0026rsquo;t identify themselves; behavioral enforcement affects them precisely because their traffic is indistinguishable from the automated traffic it targets.\nMaking Private Docs Agent-Accessible #This section offers non-normative guidance for organizations that gate their documentation. The options below are ordered roughly by implementation effort, from lowest to highest.\n1. Ungating reference documentation. The simplest option: make API references, SDK docs, and integration guides public while keeping truly sensitive content (internal architecture, security configurations, pricing tiers) behind auth. Many enterprises already do this for developer experience reasons. Agents benefit from the same split.\n2. Shipping docs with the product. Include documentation as local files in your SDK, package, or CLI tool. A docs/ directory with markdown files, comprehensive README content, or built-in help text is always available to agents reading the local filesystem. This is particularly effective for API clients and libraries where the docs are version-specific anyway.\n3. Providing a public llms.txt. Even if page content is gated, a public llms.txt that describes what documentation exists and how it\u0026rsquo;s organized gives agents a map. They can tell the developer \u0026ldquo;the rate limiting docs are at /docs/api/rate-limits, but I can\u0026rsquo;t access them; could you paste the relevant section?\u0026rdquo; This is better than the agent having no idea what docs exist at all.\n4. Supporting token-based access for agent-facing endpoints. Serve llms.txt and markdown content behind API key or bearer token authentication rather than browser-based SSO. Agents and their tooling can be configured to pass static credentials, similar to how npm or pip authenticate with private registries. This preserves access control while enabling programmatic access.\n5. Building an MCP server. An MCP server gives agents structured, authenticated access to documentation through tool calls like search_docs(\u0026quot;rate limiting\u0026quot;) or get_doc(\u0026quot;api/authentication\u0026quot;). Auth credentials are configured on the server; the agent never sees them. This is the richest option because the MCP server can provide search, filtering, and context-aware responses rather than just serving raw files. It also allows fine-grained access control (different API keys could see different content tiers).\n6. Providing a CLI with doc access. If your product already has a CLI that developers authenticate with, adding a docs subcommand gives agents access through a channel the developer has already authorized. The agent calls the CLI tool; the CLI handles authentication using the developer\u0026rsquo;s existing credentials.\nOrganizations don\u0026rsquo;t need to implement all of these. A public llms.txt combined with ungated reference docs covers the most common agent use cases with minimal effort. MCP servers are for organizations that want to provide a first-class agent experience with their private documentation.\n","date":null,"permalink":"https://agentdocsspec.com/spec/web/authentication/","section":"Specifications","summary":"\u003cp\u003ePart of the \u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e. The \u003ca href=\"https://agentdocsspec.com/spec/web/#checks-summary\" target=\"_blank\" rel=\"noreferrer\"\u003eChecks Summary\u003c/a\u003e lists all checks with links to their definitions.\u003c/p\u003e\n\u003cp\u003eThese checks evaluate whether documentation is accessible to agents at all:\nwithout requiring interactive authentication, and without infrastructure-level\nbarriers aimed at automated clients. Docs behind login walls are effectively\ninvisible to coding agents, which has significant implications as agent-assisted\ndevelopment becomes a standard workflow. Bot-protection systems can produce the\nsame invisibility through a different mechanism, and often without the site\nowner realizing documentation is affected.\u003c/p\u003e","title":"Category 7: Authentication and Access"},{"content":"Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.\nIndividual checks measure discrete properties, but agent experience can degrade non-linearly when certain failures combine. A site might pass most checks individually while still being effectively inaccessible to agents because of how the failures interact. This section describes known interaction patterns that implementations should detect and surface. Implementations should evaluate these after all individual checks have completed.\nUndiscoverable Markdown #Checks involved: markdown-url-support, content-negotiation, llms-txt-directive-html, llms-txt-directive-md, llms-txt-links-markdown\nObserved behavior: A site serves markdown at .md URLs, but agents have no way to discover this capability. Without content negotiation, a directive on pages pointing to llms.txt, or .md links in llms.txt, agents default to the HTML path and never benefit from the markdown support the site provides.\nThis matters because markdown availability is one of the highest-impact improvements a site can make, but only if agents can find it. A site in this state has done the hard work of generating markdown but gets none of the benefit.\nTruncated Index #Checks involved: llms-txt-exists, llms-txt-size\nObserved behavior: A site provides llms.txt, but the file exceeds agent context limits. Agents see the first portion of the file and lose everything after the truncation point: links, structure, and entire sections become invisible. Quality assessments of the truncated portion (link resolution, coverage, markdown links) don\u0026rsquo;t reflect what agents actually experience.\nSites with large documentation sets are most likely to hit this. The spec\u0026rsquo;s progressive disclosure recommendation (splitting into a root index linking to section-level files) directly addresses this pattern.\nClient-Rendered Pages #Checks involved: rendering-strategy, page-size-html, content-start-position\nObserved behavior: Pages that rely on client-side JavaScript rendering return an empty shell to agents instead of documentation content. When this affects a portion of a site\u0026rsquo;s pages, HTML-path measurements (page size, content start position) for those pages are measuring the shell, not the actual content. Results from those checks become unreliable for affected pages.\nThis does not mean the site is entirely inaccessible. If the site also serves markdown and agents can discover it, the markdown path still works. But agents on the HTML path receive no usable content from affected pages.\nNo Viable Content Path #Checks involved: llms-txt-exists, rendering-strategy, markdown-url-support, plus the undiscoverable markdown pattern above\nObserved behavior: Agents have no effective way to access the site\u0026rsquo;s documentation. There is no llms.txt for navigation, no discoverable markdown path, and HTML responses either don\u0026rsquo;t contain rendered content or weren\u0026rsquo;t tested. This is the lowest possible agent accessibility state.\nThis pattern represents a complete access failure rather than a degraded experience. The single highest-impact action is creating an llms.txt at the site root. If the site uses client-side rendering, enabling server-side rendering is the second priority.\nAuthenticated Docs Without Alternatives #Checks involved: auth-gate-detection, auth-alternative-access\nObserved behavior: The site\u0026rsquo;s documentation requires authentication, and no alternative access paths were detected. Agents that encounter the docs fall back on training data or seek secondary sources that may be inaccurate or outdated.\nAuthentication is a legitimate choice for many documentation sites. This pattern is notable because it means agents have no path to current content at all. Even partial alternatives (a public llms.txt as a navigational index, ungated API references, docs shipped with the SDK/package) significantly improve the agent experience compared to a complete access barrier.\nBot Protection Degrading Scan Reliability #Checks involved: bot-protection-interference, plus every multi-page check\nObserved behavior: Behavioral bot enforcement engages partway through a scan. Requests that would have succeeded in isolation begin to stall, get challenged, or fail, and every check still running is now scoring whatever sample survives. The site\u0026rsquo;s scores can look reasonable while being computed from a fraction of the intended pages.\nThis pattern has two victims. Agents doing multi-page reading sessions lose access mid-session, which is the site-side problem the check exists to surface. And the assessment itself degrades: per-check \u0026ldquo;failed to fetch\u0026rdquo; counts are scattered and easy to miss, so implementations should aggregate fetch failures at run level and flag results prominently when the failure rate is high (for example, above 20% of page fetches). A flagged run is still useful evidence; it just measures a smaller sample than it appears to.\nOversized Pages Without Markdown Escape #Checks involved: page-size-html, markdown-url-support, plus the undiscoverable markdown pattern above\nObserved behavior: Pages exceed agent context limits on the HTML path, and there is no discoverable markdown path for agents to get smaller representations. Agents receive truncated content on these pages with no warning and no alternative available.\nWhen pages are large but markdown is available and discoverable, agents that support content negotiation or follow llms.txt directives can access smaller representations. Without that escape hatch, truncation is unavoidable.\nDynamic Content Rendered Statically #Checks involved: markdown-content-parity, single-fetch-completeness, markdown-link-portability, embedded-data-serialization, plus the Category 3 size checks\nObserved behavior: A page whose content is dynamic (a filterable catalog, a data-driven matrix, a widget-rendered listing) is flattened into static markdown for agents, and the flattening fails in several ways at once. There are four characteristic failure directions: too much (widget data dumped wholesale into the content), too little (UI pagination inherited into a format that didn\u0026rsquo;t need it), inconsistent (default filters or staleness making representations disagree), and unnavigable (generated links that assume a browser context, or that are broken wholesale by the generation pipeline).\nOne observed production catalog page exhibited all four simultaneously: the HTML showed 98 items under a default filter while the markdown listed 102, the markdown was paginated with a trailing note and a continuation URL that returned an empty body, and every entry link pointed into a wrong generated path prefix that soft-404ed. Each individual check would flag one symptom; the underlying cause is shared. The markdown variant is a second rendering pipeline, and it needs the same QA the HTML pipeline gets. Implementations that detect several of these failures on the same generated page should present them as one pipeline problem rather than four independent findings.\n","date":null,"permalink":"https://agentdocsspec.com/spec/web/interaction-effects/","section":"Specifications","summary":"\u003cp\u003ePart of the \u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e. The \u003ca href=\"https://agentdocsspec.com/spec/web/#checks-summary\" target=\"_blank\" rel=\"noreferrer\"\u003eChecks Summary\u003c/a\u003e lists all checks with links to their definitions.\u003c/p\u003e\n\u003cp\u003eIndividual checks measure discrete properties, but agent experience can degrade\nnon-linearly when certain failures combine. A site might pass most checks\nindividually while still being effectively inaccessible to agents because of how\nthe failures interact. This section describes known interaction patterns that\nimplementations should detect and surface. Implementations should evaluate these\nafter all individual checks have completed.\u003c/p\u003e","title":"Interaction Effects"},{"content":"Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.\nAppendix A: Known Platform Truncation Limits #The thresholds used in this spec\u0026rsquo;s pass/warn/fail levels are derived from observed and documented platform behavior. This appendix tracks known limits so that implementations can calibrate their thresholds appropriately, and so that the spec\u0026rsquo;s default thresholds can be updated as more data becomes available.\nThresholds Used in This Spec #The spec uses two threshold tiers across its size-related checks:\n50,000 characters: The \u0026ldquo;pass\u0026rdquo; threshold. Content under this size fits comfortably within all known platform limits. 100,000 characters: The \u0026ldquo;fail\u0026rdquo; threshold. Content over this size will be truncated by Claude Code and likely by most other platforms. These are conservative defaults based on the best-documented platform (Claude Code). Implementations should allow these thresholds to be configurable so users can evaluate against specific platform limits or adjust as new data becomes available.\npage-size-transfer uses byte thresholds (1MB warn, 10MB fail) rather than character thresholds, because it measures the response before any processing. The 10MB fail line is anchored to Claude Code\u0026rsquo;s documented fetch buffer; byte-level caps on other platforms are less documented than character-level truncation limits, so these defaults are conservative and should likewise be configurable.\nKnown Platform Limits #Compare platform architecture and truncation limits in Platforms.\nWhat This Means for Threshold Selection #The MCP Fetch reference server\u0026rsquo;s default of 5,000 characters is worth noting. Many agent setups use MCP-based fetch tools, and if users haven\u0026rsquo;t changed the default, they\u0026rsquo;re working with a limit 20x smaller than Claude Code\u0026rsquo;s. A page that passes at the 50K threshold may still be unusable for MCP Fetch users with default settings.\nImplementations may want to support named profiles (e.g., --profile claude-code, --profile mcp-default) that set thresholds to match specific platforms, in addition to allowing custom threshold values.\nAppendix B: Notable Exclusions #This section documents topics that were considered for the spec but intentionally excluded, along with the rationale.\nrobots.txt and AI User-Agent Blocking #robots.txt can block known AI training crawlers (ClaudeBot, GPTBot, Google-Extended, etc.) that identify themselves via user-agent strings. However, this is a crawling policy concern rather than an agent-friendliness concern, and the two audiences are distinct.\nTraining crawlers and coding agents are different request paths with different user-agents. The agents this spec targets (coding assistants fetching docs during real-time workflows) are largely invisible to robots.txt:\nAgent User-Agent Identifiable as AI? Claude Code axios/1.8.4 No (generic HTTP library) Cursor Standard Chrome UA No OpenCode Standard Chrome UA No GitHub Copilot Electron/VS Code UA No (looks like normal IDE traffic) OpenAI Codex ChatGPT-User/1.0 Yes Gemini CLI GoogleAgent-URLContext Yes Windsurf colly Somewhat (Go scraping library) Source: Checkly, \u0026ldquo;State of AI Agent Content Negotiation\u0026rdquo;\nMost coding agents use standard browser user-agent strings and are indistinguishable from human traffic. A site blocking ClaudeBot in robots.txt is blocking Anthropic\u0026rsquo;s training crawler, not Claude Code fetching a docs page. Since this spec is about making documentation accessible to agents in real-time workflows, robots.txt configuration is out of scope.\nBehavioral bot protection is a different matter. Because most coding agents don\u0026rsquo;t identify themselves, bot-management systems that act on traffic behavior rather than declared identity affect them even when crawling policy does not. That failure mode is in scope; see bot-protection-interference.\nGitHub Raw URL Fallback #GitHub raw URLs (raw.githubusercontent.com/...) were observed to be the single most reliable documentation access pattern in practice. When official docs failed (rate-limited, JavaScript-rendered, or hard to navigate), GitHub was almost always a viable fallback.\nHowever, this is a fallback strategy for agent users rather than a property of the documentation site itself. Whether a project\u0026rsquo;s docs source happens to be on GitHub, and whether the raw content there is usable as standalone documentation, is outside the control of a docs site evaluation. This spec focuses on what documentation site owners can do to improve agent accessibility of their own sites.\n","date":null,"permalink":"https://agentdocsspec.com/spec/web/appendices/","section":"Specifications","summary":"\u003cp\u003ePart of the \u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e. The \u003ca href=\"https://agentdocsspec.com/spec/web/#checks-summary\" target=\"_blank\" rel=\"noreferrer\"\u003eChecks Summary\u003c/a\u003e lists all checks with links to their definitions.\u003c/p\u003e\n\u003ch2 id=\"appendix-a-known-platform-truncation-limits\" class=\"relative group\"\u003eAppendix A: Known Platform Truncation Limits \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#appendix-a-known-platform-truncation-limits\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h2\u003e\u003cp\u003eThe thresholds used in this spec\u0026rsquo;s pass/warn/fail levels are derived from\nobserved and documented platform behavior. This appendix tracks known limits\nso that implementations can calibrate their thresholds appropriately, and so\nthat the spec\u0026rsquo;s default thresholds can be updated as more data becomes\navailable.\u003c/p\u003e","title":"Appendices"},{"content":"Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.\nv0.6.0 (2026-09-13) # Added bot-protection-interference (Category 7: Authentication and Access). Detects bot-protection systems interfering with automated documentation fetching: challenge interstitials served as 200, tarpits that return headers and then stall the response body indefinitely, and volume-triggered throttling or blocking. Grounded in an observed production case where CDN bot management responded to a sustained scan by holding response bodies open; single-request probes looked healthy throughout, and enforcement decayed after a cooldown. Detection is heuristic, observed as a byproduct of a normal scan rather than a directed probe. Added the Bot Protection Degrading Scan Reliability interaction effect: when enforcement engages mid-scan, other checks score the surviving sample, so implementations should aggregate fetch failures at run level and flag results when the failure rate is high. Expanded the Category 7 introduction to cover infrastructure-level access barriers alongside authentication. Added page-size-transfer (Category 3: Page Size and Truncation Risk). Measures the served byte size of the HTML document response, a failure mode page-size-html\u0026rsquo;s post-conversion measurement cannot see. Grounded in measurements of production documentation sites where 75-84% of page bytes were serialized framework payloads (component trees, resolved metadata, embedded duplicate markdown source), with served-bytes-to-content ratios from 40:1 to 200:1. Scored on served bytes; the ratio is reported as a diagnostic. Documented the pipeline distinction in Category 3\u0026rsquo;s \u0026ldquo;How Agents Get Content\u0026rdquo; section: convert-then-truncate, truncate-then-convert (or raw ingestion), and capped fetch, and which size measurement predicts each. Added single-fetch-completeness (Category 3: Page Size and Truncation Risk). Detects pagination in markdown responses and verifies the continuation is machine-followable. Grounded in an observed production catalog whose markdown variant showed 100 of 102 entries with a trailing relative continuation URL that returned an empty body, while the complete set would have fit well under the 50,000-character pass threshold. Added markdown-link-portability (Category 4: Content Structure). Links in served markdown should be absolute and should resolve to the representation they promise, verified beyond status codes (content type and soft-404 heuristics). Grounded in an observed production catalog whose generated markdown links all pointed into a wrong path prefix and soft-404ed as HTML SPA shells at .md URLs while returning 200. Added embedded-data-serialization (Category 4: Content Structure). Attributes page size to machine-generated bulk elements (large uniform tables, inline data blobs). Grounded in a measured production reference page: 302KB of HTML, 64% table markup including a single 218-row generated table, converting to ~83,000 characters while its non-table prose totaled ~17,000. Extended markdown-content-parity notes with guidance for dynamically generated pages: compare item counts between representations and distinguish default-filter divergence, pagination windowing, and staleness as causes. Added the Dynamic Content Rendered Statically interaction effect: the four characteristic ways a dynamic page flattens badly into static markdown (too much, too little, inconsistent, unnavigable), observed together on a single production catalog page. Added the Related Surfaces subsection to Scope, naming the planned companion specifications (content composition, repository-local documentation) and the boundary that keeps this spec\u0026rsquo;s checks mechanically verifiable. Added the Serving RAG Ingestion Pipelines informational section, mapping existing checks to RAG ingestion needs (crawl manifest, clean source, chunk boundaries, incremental re-indexing), informed by consumer reports from production RAG builds. The Scope section\u0026rsquo;s RAG exclusion now distinguishes query-time retrieval (out of scope) from ingestion (served by this spec). Restructured the website serving of the spec: the full document exceeded the 100,000-character truncation threshold its own checks warn about, so it is now served as per-category pages under /spec/web/, with /spec/ becoming a landing page for this and future companion specifications. The canonical source remains a single SPEC.md in the repository. This breaks previously published deep URLs deliberately, in exchange for a namespace that accommodates the companion specs. Check count: 23 → 28. v0.5.1 (2026-05-08) # Moved per-platform truncation data out of Appendix A into a new Platforms comparison page on the site. Appendix A retains the spec\u0026rsquo;s threshold rationale and points readers to the platforms page for current per-platform observations. Category 3 (Page Size and Truncation Risk) now references the platforms page so readers can connect threshold choices to empirical pipeline behavior. No threshold or check definitions changed. Platforms page authored by Rhyannon Rodriguez. v0.5.0 (2026-04-25) # Split llms-txt-directive into two independent checks: llms-txt-directive-html and llms-txt-directive-md. The original check conflated two distinct signals that serve different audiences. The HTML check detects directives in the rendered DOM (for agents fetching HTML pages); the markdown check detects directives in markdown source (for agents fetching .md URLs or using content negotiation). The split also adds explicit detection guidance: incidental mentions of llms.txt in navigation, metadata, or page content discussing the feature do not count as directives. llms-txt-directive-md depends on markdown-url-support or content-negotiation; it is skipped if neither passes. This is a breaking change for implementations that reference the old check ID. Check count: 22 → 23. v0.4.0 (2026-04-21) # Renamed llms-txt-freshness to llms-txt-coverage. The check compares llms.txt URLs against the sitemap to measure how much of the site is represented; that\u0026rsquo;s coverage, not freshness. Whether listed URLs still resolve is already handled by llms-txt-links-resolve. Rewrote the check description to match. This is a breaking change for implementations that reference the old check ID. Revised page-size-html and content-start-position to be pipeline-agnostic. The previous language prescribed a specific conversion approach (Turndown with default configuration) based on one agent\u0026rsquo;s behavior. Agent HTML processing pipelines vary and continue to evolve; the spec now describes the measurement goal (approximate what agents see) and leaves conversion details to implementers. Recommended actions now cover all boilerplate sources (navigation, sidebars, serialized tabbed content) rather than focusing narrowly on inline CSS/JS. Expanded llms-txt-coverage to account for intentional curation. Many sites deliberately include only a subset of pages in llms.txt (excluding changelogs, old versions, directory pages, etc.). The check now describes three use cases (full parity, curated, hybrid) served by configurable thresholds and exclusion patterns, rather than treating all gaps as problems. Expanded markdown-content-parity to distinguish intentional audience segmentation from unintentional content drift. Some sites intentionally serve different content per audience (agent-optimized markdown vs. human-optimized HTML). The check now describes audience-segmentation tags as a mechanism implementations can recognize, and supports the same mirrored/segmented/curated spectrum as llms-txt-coverage. The spec does not prescribe specific tag conventions; implementations document which they support. v0.3.0 (2026-03-31) # Merged Category 6 (Agent Discoverability Directives) into Category 1, renamed to \u0026ldquo;Content Discoverability.\u0026rdquo; The llms-txt-directive check (now llms-txt-directive-html and llms-txt-directive-md) answers the same fundamental question as the llms.txt checks: can agents find and navigate the content? This reduces categories from 8 to 7. Renumbered Category 7 (Observability) to 6, Category 8 (Authentication) to 7. Added Recommended action field to all 22 check definitions. Provides 1-2 sentence actionable guidance for each warn and fail state, giving documentation teams a clear next step rather than just a diagnosis. Added Interaction Effects section after Checks Summary. Documents six patterns where combinations of check results indicate systemic problems worse than individual failures suggest (e.g., undiscoverable markdown, no viable content path, oversized pages without markdown escape). Category count: 8 → 7. Check count unchanged at 22. v0.2.1 (2026-03-15) #Clarifications from implementing the afdocs conformance tool against the spec. No new checks; all changes refine existing check definitions.\nmarkdown-code-fence-validity: Removed warn level. Per CommonMark, mismatched delimiters (opening ``` closing ~~~) produce unclosed fences, not a distinct \u0026ldquo;mismatched but balanced\u0026rdquo; state. The described warn case was indistinguishable from a fail. llms-txt-directive (now llms-txt-directive-html and llms-txt-directive-md): Clarified that pass requires the directive in all (or nearly all) pages, not just presence in any single page. Clarified warn triggers: missing from some pages, or present but buried past 50%. llms-txt-freshness (now llms-txt-coverage): Added default thresholds (\u0026gt;=95% pass, 80-95% warn, \u0026lt;80% fail) for sitemap coverage. The previous language was qualitative; implementations need concrete defaults for automation. markdown-content-parity: Added default thresholds (\u0026lt;5% missing pass, 5-20% warn, \u0026gt;=20% fail) for content segment comparison. Same rationale. section-header-quality: Added default thresholds (\u0026lt;=25% generic pass, 25-50% warn, \u0026gt;50% fail) and clarified that evaluation covers both within-group and cross-group header repetition. cache-header-hygiene: Added exception for responses with ETag or Last-Modified but no Cache-Control/Expires. These validation headers enable conditional revalidation and should not be penalized. rendering-strategy: Clarified that the note about downstream checks (page-size-html, content-start-position) being unreliable is guidance for report consumers, not an implementation dependency requirement. v0.2.0 (2026-03-15) # New check: rendering-strategy (Category 3). Detects pages that rely on client-side JavaScript to render content, which makes them invisible to most coding agents. Covers full SPA shells and the subtler case of statically generated pages with client-side content population. Check count: 21 → 22. v0.1.0 (2026-02-22) - Initial Draft # Initial spec with 21 checks across 8 categories. Progressive disclosure recommendation for large llms.txt files. Authentication and access category: auth gate detection, alternative access paths, and guidance for making private docs agent-accessible. Known platform truncation limits (Appendix A). Notable exclusions with rationale (Appendix B). ","date":null,"permalink":"https://agentdocsspec.com/spec/web/changelog/","section":"Specifications","summary":"\u003cp\u003ePart of the \u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e. The \u003ca href=\"https://agentdocsspec.com/spec/web/#checks-summary\" target=\"_blank\" rel=\"noreferrer\"\u003eChecks Summary\u003c/a\u003e lists all checks with links to their definitions.\u003c/p\u003e\n\u003ch3 id=\"v060-2026-09-13\" class=\"relative group\"\u003ev0.6.0 (2026-09-13) \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#v060-2026-09-13\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003eAdded \u003ccode\u003ebot-protection-interference\u003c/code\u003e (Category 7: Authentication and Access).\nDetects bot-protection systems interfering with automated documentation\nfetching: challenge interstitials served as 200, tarpits that return headers\nand then stall the response body indefinitely, and volume-triggered\nthrottling or blocking. Grounded in an observed production case where CDN\nbot management responded to a sustained scan by holding response bodies\nopen; single-request probes looked healthy throughout, and enforcement\ndecayed after a cooldown. Detection is heuristic, observed as a byproduct\nof a normal scan rather than a directed probe.\u003c/li\u003e\n\u003cli\u003eAdded the \u003ca href=\"https://agentdocsspec.com/spec/web/interaction-effects/#bot-protection-degrading-scan-reliability\" target=\"_blank\" rel=\"noreferrer\"\u003eBot Protection Degrading Scan Reliability\u003c/a\u003e\ninteraction effect: when enforcement engages mid-scan, other checks score\nthe surviving sample, so implementations should aggregate fetch failures at\nrun level and flag results when the failure rate is high.\u003c/li\u003e\n\u003cli\u003eExpanded the Category 7 introduction to cover infrastructure-level access\nbarriers alongside authentication.\u003c/li\u003e\n\u003cli\u003eAdded \u003ccode\u003epage-size-transfer\u003c/code\u003e (Category 3: Page Size and Truncation Risk).\nMeasures the served byte size of the HTML document response, a failure\nmode \u003ccode\u003epage-size-html\u003c/code\u003e\u0026rsquo;s post-conversion measurement cannot see. Grounded\nin measurements of production documentation sites where 75-84% of page\nbytes were serialized framework payloads (component trees, resolved\nmetadata, embedded duplicate markdown source), with served-bytes-to-content\nratios from 40:1 to 200:1. Scored on served bytes; the ratio is reported\nas a diagnostic.\u003c/li\u003e\n\u003cli\u003eDocumented the pipeline distinction in Category 3\u0026rsquo;s \u0026ldquo;How Agents Get\nContent\u0026rdquo; section: convert-then-truncate, truncate-then-convert (or raw\ningestion), and capped fetch, and which size measurement predicts each.\u003c/li\u003e\n\u003cli\u003eAdded \u003ccode\u003esingle-fetch-completeness\u003c/code\u003e (Category 3: Page Size and Truncation\nRisk). Detects pagination in markdown responses and verifies the\ncontinuation is machine-followable. Grounded in an observed production\ncatalog whose markdown variant showed 100 of 102 entries with a trailing\nrelative continuation URL that returned an empty body, while the complete\nset would have fit well under the 50,000-character pass threshold.\u003c/li\u003e\n\u003cli\u003eAdded \u003ccode\u003emarkdown-link-portability\u003c/code\u003e (Category 4: Content Structure). Links\nin served markdown should be absolute and should resolve to the\nrepresentation they promise, verified beyond status codes (content type\nand soft-404 heuristics). Grounded in an observed production catalog\nwhose generated markdown links all pointed into a wrong path prefix and\nsoft-404ed as HTML SPA shells at \u003ccode\u003e.md\u003c/code\u003e URLs while returning 200.\u003c/li\u003e\n\u003cli\u003eAdded \u003ccode\u003eembedded-data-serialization\u003c/code\u003e (Category 4: Content Structure).\nAttributes page size to machine-generated bulk elements (large uniform\ntables, inline data blobs). Grounded in a measured production reference\npage: 302KB of HTML, 64% table markup including a single 218-row\ngenerated table, converting to ~83,000 characters while its non-table\nprose totaled ~17,000.\u003c/li\u003e\n\u003cli\u003eExtended \u003ccode\u003emarkdown-content-parity\u003c/code\u003e notes with guidance for dynamically\ngenerated pages: compare item counts between representations and\ndistinguish default-filter divergence, pagination windowing, and\nstaleness as causes.\u003c/li\u003e\n\u003cli\u003eAdded the \u003ca href=\"https://agentdocsspec.com/spec/web/interaction-effects/#dynamic-content-rendered-statically\" target=\"_blank\" rel=\"noreferrer\"\u003eDynamic Content Rendered Statically\u003c/a\u003e\ninteraction effect: the four characteristic ways a dynamic page flattens\nbadly into static markdown (too much, too little, inconsistent,\nunnavigable), observed together on a single production catalog page.\u003c/li\u003e\n\u003cli\u003eAdded the \u003ca href=\"https://agentdocsspec.com/spec/web/#related-surfaces\" target=\"_blank\" rel=\"noreferrer\"\u003eRelated Surfaces\u003c/a\u003e subsection to Scope,\nnaming the planned companion specifications (content composition,\nrepository-local documentation) and the boundary that keeps this spec\u0026rsquo;s\nchecks mechanically verifiable.\u003c/li\u003e\n\u003cli\u003eAdded the \u003ca href=\"https://agentdocsspec.com/spec/web/#serving-rag-ingestion-pipelines\" target=\"_blank\" rel=\"noreferrer\"\u003eServing RAG Ingestion Pipelines\u003c/a\u003e\ninformational section, mapping existing checks to RAG ingestion needs\n(crawl manifest, clean source, chunk boundaries, incremental\nre-indexing), informed by consumer reports from production RAG builds.\nThe Scope section\u0026rsquo;s RAG exclusion now distinguishes query-time retrieval\n(out of scope) from ingestion (served by this spec).\u003c/li\u003e\n\u003cli\u003eRestructured the website serving of the spec: the full document exceeded\nthe 100,000-character truncation threshold its own checks warn about, so\nit is now served as per-category pages under \u003ccode\u003e/spec/web/\u003c/code\u003e, with \u003ccode\u003e/spec/\u003c/code\u003e\nbecoming a landing page for this and future companion specifications.\nThe canonical source remains a single SPEC.md in the repository. This\nbreaks previously published deep URLs deliberately, in exchange for a\nnamespace that accommodates the companion specs.\u003c/li\u003e\n\u003cli\u003eCheck count: 23 → 28.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"v051-2026-05-08\" class=\"relative group\"\u003ev0.5.1 (2026-05-08) \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#v051-2026-05-08\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003eMoved per-platform truncation data out of Appendix A into a new\n\u003ca href=\"https://agentdocsspec.com/platforms/\" target=\"_blank\" rel=\"noreferrer\"\u003ePlatforms\u003c/a\u003e comparison page on the\nsite. Appendix A retains the spec\u0026rsquo;s threshold rationale and points readers\nto the platforms page for current per-platform observations. Category 3\n(Page Size and Truncation Risk) now references the platforms page so\nreaders can connect threshold choices to empirical pipeline behavior. No\nthreshold or check definitions changed. Platforms page authored by\nRhyannon Rodriguez.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"v050-2026-04-25\" class=\"relative group\"\u003ev0.5.0 (2026-04-25) \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#v050-2026-04-25\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003eSplit \u003ccode\u003ellms-txt-directive\u003c/code\u003e into two independent checks:\n\u003ccode\u003ellms-txt-directive-html\u003c/code\u003e and \u003ccode\u003ellms-txt-directive-md\u003c/code\u003e. The original check\nconflated two distinct signals that serve different audiences. The HTML\ncheck detects directives in the rendered DOM (for agents fetching HTML\npages); the markdown check detects directives in markdown source (for\nagents fetching \u003ccode\u003e.md\u003c/code\u003e URLs or using content negotiation). The split also\nadds explicit detection guidance: incidental mentions of \u003ccode\u003ellms.txt\u003c/code\u003e in\nnavigation, metadata, or page content discussing the feature do not count\nas directives. \u003ccode\u003ellms-txt-directive-md\u003c/code\u003e depends on \u003ccode\u003emarkdown-url-support\u003c/code\u003e\nor \u003ccode\u003econtent-negotiation\u003c/code\u003e; it is skipped if neither passes. This is a\nbreaking change for implementations that reference the old check ID.\u003c/li\u003e\n\u003cli\u003eCheck count: 22 → 23.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"v040-2026-04-21\" class=\"relative group\"\u003ev0.4.0 (2026-04-21) \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#v040-2026-04-21\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003eRenamed \u003ccode\u003ellms-txt-freshness\u003c/code\u003e to \u003ccode\u003ellms-txt-coverage\u003c/code\u003e. The check compares\n\u003ccode\u003ellms.txt\u003c/code\u003e URLs against the sitemap to measure how much of the site is\nrepresented; that\u0026rsquo;s coverage, not freshness. Whether listed URLs still\nresolve is already handled by \u003ccode\u003ellms-txt-links-resolve\u003c/code\u003e. Rewrote the check\ndescription to match. This is a breaking change for implementations that\nreference the old check ID.\u003c/li\u003e\n\u003cli\u003eRevised \u003ccode\u003epage-size-html\u003c/code\u003e and \u003ccode\u003econtent-start-position\u003c/code\u003e to be\npipeline-agnostic. The previous language prescribed a specific conversion\napproach (Turndown with default configuration) based on one agent\u0026rsquo;s\nbehavior. Agent HTML processing pipelines vary and continue to evolve;\nthe spec now describes the measurement goal (approximate what agents see)\nand leaves conversion details to implementers. Recommended actions now\ncover all boilerplate sources (navigation, sidebars, serialized tabbed\ncontent) rather than focusing narrowly on inline CSS/JS.\u003c/li\u003e\n\u003cli\u003eExpanded \u003ccode\u003ellms-txt-coverage\u003c/code\u003e to account for intentional curation. Many\nsites deliberately include only a subset of pages in \u003ccode\u003ellms.txt\u003c/code\u003e (excluding\nchangelogs, old versions, directory pages, etc.). The check now describes\nthree use cases (full parity, curated, hybrid) served by configurable\nthresholds and exclusion patterns, rather than treating all gaps as\nproblems.\u003c/li\u003e\n\u003cli\u003eExpanded \u003ccode\u003emarkdown-content-parity\u003c/code\u003e to distinguish intentional audience\nsegmentation from unintentional content drift. Some sites intentionally\nserve different content per audience (agent-optimized markdown vs.\nhuman-optimized HTML). The check now describes audience-segmentation tags\nas a mechanism implementations can recognize, and supports the same\nmirrored/segmented/curated spectrum as \u003ccode\u003ellms-txt-coverage\u003c/code\u003e. The spec does\nnot prescribe specific tag conventions; implementations document which\nthey support.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"v030-2026-03-31\" class=\"relative group\"\u003ev0.3.0 (2026-03-31) \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#v030-2026-03-31\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cul\u003e\n\u003cli\u003eMerged Category 6 (Agent Discoverability Directives) into Category 1,\nrenamed to \u0026ldquo;Content Discoverability.\u0026rdquo; The \u003ccode\u003ellms-txt-directive\u003c/code\u003e check (now\n\u003ccode\u003ellms-txt-directive-html\u003c/code\u003e and \u003ccode\u003ellms-txt-directive-md\u003c/code\u003e) answers\nthe same fundamental question as the llms.txt checks: can agents find and\nnavigate the content? This reduces categories from 8 to 7.\u003c/li\u003e\n\u003cli\u003eRenumbered Category 7 (Observability) to 6, Category 8 (Authentication) to 7.\u003c/li\u003e\n\u003cli\u003eAdded \u003cstrong\u003eRecommended action\u003c/strong\u003e field to all 22 check definitions. Provides\n1-2 sentence actionable guidance for each warn and fail state, giving\ndocumentation teams a clear next step rather than just a diagnosis.\u003c/li\u003e\n\u003cli\u003eAdded \u003cstrong\u003eInteraction Effects\u003c/strong\u003e section after Checks Summary. Documents six\npatterns where combinations of check results indicate systemic problems\nworse than individual failures suggest (e.g., undiscoverable markdown,\nno viable content path, oversized pages without markdown escape).\u003c/li\u003e\n\u003cli\u003eCategory count: 8 → 7. Check count unchanged at 22.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch3 id=\"v021-2026-03-15\" class=\"relative group\"\u003ev0.2.1 (2026-03-15) \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#v021-2026-03-15\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h3\u003e\u003cp\u003eClarifications from implementing the \u003ccode\u003eafdocs\u003c/code\u003e conformance tool against the\nspec. No new checks; all changes refine existing check definitions.\u003c/p\u003e","title":"Changelog"},{"content":" Author Rhyannon Rodriguez Last updated 2026-05-08 Methodology Agent Ecosystem Testing This page provides an overview of observed agent web fetch retrieval behavior across agent platforms. To clarify observed behavior, we break down the retrieval process into three key components that form most agent web fetch pipelines:\nRetrieval: How and when an agent fetches content Truncation: What gets lost and whether agents report it Summarization: What happens to content between retrieval and generation These observations inform the size thresholds and pipeline assumptions in the Web Documentation Delivery Spec, particularly Category 3: Page Size and Truncation Risk.\nRetrieval #The web fetch gap isn\u0026rsquo;t in retrieval, but in what follows: how agents attend to various content types during generation, whether that\u0026rsquo;s context window handling, chunking losses, or summarization. Platform links lead to each tool\u0026rsquo;s official documentation.\nPlatform Prompt Syntax Invocation Pattern Retrieval Behavior Claude API web fetch Enable tool to augment Claude\u0026rsquo;s context with URL Mid-generation deterministic: tool requires enablement in API request, includes URL validation and results cache, may or may not provide live web content Visibility high: only platform where response body includes raw tool result; no JavaScript execution, CSS-heavy pages and/or SPAs often return little to no prose Claude Code WebFetch invoked automatically with prompt URL Mid-generation deterministic: returns cached result if available, otherwise fetches live Visibility high: Markdown result returned directly if trusted, text/markdown, \u0026lt;100k; otherwise a smaller LLM extracts relevant content before passing to Claude; no JavaScript execution Cursor No web fetch behavior publicly documented, @Web context attachment redundant, agents don\u0026rsquo;t correct misuse Mid-generation nondeterministic: Auto default setting autonomous LLM and fetch method selection per request Visibility low: fetch method not explicitly named, no JavaScript execution, CSS-heavy pages and/or SPAs often return little to no prose; prefers Markdown, content negotation documented with Accept: text/markdown; sends full browser fingerprint: User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/139.0.0.0 Safari/537.36 with Chrome client hints Sec-Ch-Ua, Sec-Fetch-* Gemini API URL context Enable tool to augment Gemini\u0026rsquo;s context with URL, request requires url_context with full, unnested URLs Pre-generation injection deterministic: two-step process, fetches from internal cache, if unsuccessful, then live fetch; documentation includes parsing limitations Visibility low: retrieved content injected into context without a testable field, retrieval orchestration and generation process opaque; url_context_metadata order nondeterministic, authoritative signal url_retrieval_status, tool_use_prompt_token_count only size proxy GitHub Copilot No web fetch behavior publicly documented, prompt with URL Mid-generation nondeterministic: Auto default setting autonomous LLM and fetch method selection per request Visibility medium: intermittently tools named via error, fetch_webpage returns relevance-ranked excerpts with elision markers, occasional nonlinear, inaccurate reassembly, curl byte-perfect retrieval, but no prose; content negotiation tool-dependent, presents as a browser, but overclaims User-Agent: Mozilla/5.0, AppleWebKit Accept: full HTML, curl/8.7.1 no preference,Accept: */* MCP Fetch (reference server) url required; max_length, start_index, raw optional Mid-generation deterministic: fetch invoked automatically with URL Visibility high: returns extracted contents as Markdown; supports chunked reading via start_index, allowing LLM to page through content until it finds what\u0026rsquo;s needed; no JavaScript execution OpenAI web search Chat Completions API augments GPT\u0026rsquo;s search with URL, Responses API for web_search Mid-generation nondeterministic: integration and agent-dependent: static facts and trivial math don\u0026rsquo;t invoke the tool; Chat Completions search implicit, Responses web_search_preview conditional, control cached/indexed or live content external_web_access Visibility low: Responses response.output\u0026rsquo;s web_search_call names tools, but search context not equal to LLM context window; no JavaScript execution; search_context_size: low/medium/high controls context amount, Chat Completions latency lever consistent, but Responses inconsistent Windsurf Cascade Web and docs search partially documented, @web directive redundant with URL, agents don\u0026rsquo;t correct misuse Mid-generation deterministic: autonomous two-stage pipeline designed to emulate human browsing and skimming, documentation acknowledges not all pages parseable Visibility medium: read_url_content returns chunk index with summaries, metadata and requires sequential view_content_chunk calls; curl substitution for CSS-heavy pages, SPAs return ~20–35% of expected size, little or no prose; agents used @web\u0026rsquo;s web_search as verification once every ~60 turns; presentation transparent about using crawler-scaper, but underdelivers, User-Agent: Colly Truncation #Pipelines are lossy by design in attempt to balance token cost, speed, and access to fresh content. Agents intermittently acknowledge architectural constraints, misattribute truncation causes, or self-report completeness when content is incomplete or unusable. Platform links lead to empirical testing analysis and/or tool documentation.\nPlatform Truncation Limit Observations Claude API web fetch ~20,700 chars and/or ~100 KB of rendered content default unset max_content_tokens approximate, setting 5,000 returned 17,186 chars, truncation occurs mid-token. Default limit identified in raw track, self-report attributed missing content to JavaScript rendering, masking character limit. Claude Code ~100,000 chars Trusted sites serving text/markdown under 100K chars bypass summarization, while content over 100K chars are passed to a summarization LLM. Cursor 28 KB–240 KB+ method-dependent, nondeterministic filtering WebFetch MCP ~28 KB, urllib ~72 KB, unknown path 245 KB+, curl no ceiling detected; appears to apply structure-aware content filtering, navigation and CSS stripped, but content selection heuristic presents as complete, so agents don\u0026rsquo;t report truncation. Gemini API URL context No fixed ceiling or silent dropping detected, 20 URLs hard limit per request API-layer rejection returns 400 and doesn\u0026rsquo;t consume tokens; retrieval-layer failure completes the request, but records URL_RETRIEVAL_STATUS_ERROR. Format support inconsistent with documentation: PDF fails, YouTube succeeds, JSON nondeterministic; Google Docs fail consistently. GitHub Copilot No fixed ceiling detected, nondeterministic excerpting, tested 6.68M tokens Pipeline with fetch_webpage discards whole sections or more granularly before generation, curl delivers all raw bytes but unreadable, chat rendering cutoff visible in output, not persisted as requested, but agents don\u0026rsquo;t reliably report these results as truncation. MCP Fetch (reference server) Default 5,000 chars Default max_length is 5,000 chars, but configurable up to 1,000,000; uniquely user-controlled truncation. OpenAI web search No fixed ceiling or silent dropping detected Raw source count stable at 12 regardless of search_context_size setting. Query construction not temporally aware, internal queries append training-era date strings despite running in 2026. Documented domain filtering limits not functional in Python SDK. Windsurf Cascade No fixed ceiling detected at retrieval stage, nondeterministic agent-dependent write ceiling Full retrieval agent and doc-size-dependent. Agents often retrieve fully under ~14 chunks, spotty at ~35, sparse sampling at 50+. Chunk index summary population not guaranteed, those present often include byte-count loss notices. Unique read-write asymmetry. Agents often self-report full retrieval, but fail to prove it with a write task or report truncation. Summarization #Processing layer observability vary by implementation. Platforms often offer user-configured subagents while turn-by-turn chat interactions abstract any default orchestrator-subagent relationships away. Observable outputs from default settings primarily inform the conclusions below.\nPlatform Processing Layer Inference Claude API web fetch Dynamic filtering optional, web_fetch_20260209 Server-side tool called directly with inspectable tool result in response. Dynamic filtering available with certain LLMs in which Claude writes, executes code to filter before content reaches the context window, but it\u0026rsquo;s not default behavior. Claude Code Summarization threshold-triggered Content under ~100K chars from trusted text/markdown sources reaches the context window directly without intermediate processing, but content exceeding this threshold goes through a summarization LLM that may lose information. Cursor Inferred via filtering, undocumented for web fetch Codebase research, terminal commands, and browser automation requests trigger built-in subagents explore, bash, and browser. Test prompts likely invoked explore and bash alongside web fetch. Backend routing and structure-aware content filtering suggest a pre-generation processing layer rather than a passive, linear pipeline. Gemini API URL context API layer pipeline, undocumented Pre-generation injection suggests processing occurs before LLM invocation. No transformation layer between retrieval and generation; LLM receives content directly and any summarization occurs as part of generation, not as an intermediate pipeline stage. GitHub Copilot Inferred via relevance-ranking, undocumented for web fetch Reassembled excerpts, outputs that don\u0026rsquo;t note discarded content, browser masquerading, and tool substitution patterns suggests an orchestrator-subagent relationship and not a linear, passive pipeline. Agent loop descriptions vary by implementation. VS Code-Copilot docs describe subagent delegation as main agent-initiated for complex tasks with further config available, but Copilot SDK docs only describe subagents as configurable, and not default architecture. MCP Fetch (reference server) None hard truncation at max_length Passive, linear pipeline without a processing layer. OpenAI web search Differs by API surface, undocumented Chat Completions autonomously retrieves, but Responses\u0026rsquo; LLM actively manages search in the chain of thought with open_page and find_in_page, suggesting a processing layer, but not explicitly documented or named in either API responses. Windsurf Cascade Inferred via chunking, undocumented for web and docs search Codebase research triggers built-in subagent Fast Context. Test prompts likely invoked Fast Context alongside web search. Chunk analysis, tool substitution, terminal execution, and workspace referencing suggest an extensive processing layer rather than a passive, linear pipeline. ","date":null,"permalink":"https://agentdocsspec.com/platforms/","section":"Agent platform comparisons","summary":"\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e\u003c/th\u003e\n          \u003cth\u003e\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eAuthor\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eRhyannon Rodriguez\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eLast updated\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e2026-05-08\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eMethodology\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"https://rhyannonjoy.github.io/agent-ecosystem-testing\" target=\"_blank\" rel=\"noreferrer\"\u003eAgent Ecosystem Testing\u003c/a\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003cp\u003eThis page provides an overview of observed agent web fetch retrieval behavior across agent platforms. To clarify observed behavior, we break down the retrieval process into three key components that form most agent web fetch pipelines:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003ca href=\"#retrieval\"\u003eRetrieval\u003c/a\u003e: How and when an agent fetches content\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#truncation\"\u003eTruncation\u003c/a\u003e: What gets lost and whether agents report it\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"#summarization\"\u003eSummarization\u003c/a\u003e: What happens to content between retrieval and generation\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eThese observations inform the size thresholds and pipeline assumptions in the \n      \n    \u003ca href=\"https://agentdocsspec.com/spec/web/\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e, particularly \n      \n    \u003ca href=\"https://agentdocsspec.com/spec/web/page-size/\"\u003eCategory 3: Page Size and Truncation Risk\u003c/a\u003e.\u003c/p\u003e","title":"Agent platform comparisons"},{"content":"Documentation sites are increasingly consumed by coding agents (Claude Code, Cursor, GitHub Copilot, and others) rather than human readers. But most sites weren\u0026rsquo;t built for this access pattern. Agents hit truncation limits, get walls of CSS instead of content, can\u0026rsquo;t follow cross-host redirects, and don\u0026rsquo;t know about emerging discovery mechanisms like llms.txt.\nThe result: agents frequently fail to get the documentation they need, fall back on training data, or work with partial information without knowing it.\nThe Web Documentation Delivery Spec defines 28 checks across 7 categories that evaluate how well a documentation site serves agent consumers.\nCategory Checks What it evaluates Content Discoverability 7 Discovery index exists, is valid, fits in a single fetch, links resolve, links point to markdown, embedded directives in HTML and markdown pointing agents to llms.txt Markdown Availability 2 .md URL support, content negotiation via Accept headers Page Size 6 Rendering strategy (SPA/CSR detection), markdown size, HTML size (pre/post conversion), served transfer size, content start position, single-fetch completeness Content Structure 5 Tabbed content serialization blowup, section header quality, code fence validity, markdown link portability, embedded data serialization URL Stability 2 Soft 404 detection, redirect behavior Observability 3 llms.txt coverage, markdown/HTML content parity, cache header hygiene Authentication 3 Auth gate detection, alternative access paths for gated content, bot-protection interference with automated fetching Each check has defined pass/warn/fail criteria, an automation level, and severity.\nRead the Spec\nWeb delivery is the first surface in a planned family of specifications. Companion specs for content composition (what documentation should contain to serve agents well) and repository-local documentation (docs agents grep and read inside codebases) will land at Specifications as the evidence base for them matures.\nFor empirical observations on how specific agent platforms (Claude, Cursor, Copilot, Gemini, and others) handle retrieval, truncation, and summarization, see Agent platform comparisons.\nQuick Start for Documentarians #If you can only do a few things, these have the highest impact:\nCreate an llms.txt under 50K characters. This is the single most effective discovery mechanism observed. Serve markdown versions of your pages via .md URLs or content negotiation, and verify what you serve: generated markdown can ship broken links or partial content while the HTML looks fine. Keep pages under 50K characters of content. Break up mega-pages, serialized tab content, and generated data tables. Add an llms.txt pointer to the top of every docs page. Don\u0026rsquo;t break your URLs. Use same-host HTTP redirects if you must move content. Make sure bot protection isn\u0026rsquo;t blocking agents. Exempt public docs routes from behavioral enforcement; for affected sites, this outranks everything else here. Monitor your agent-facing resources. Keep llms.txt fresh, verify markdown parity, and check cache headers. Test Your Docs #afdocs is a companion CLI tool and Node.js library that implements this spec. It runs the automated checks against your documentation site and reports what\u0026rsquo;s working, what\u0026rsquo;s not, and what to fix.\nnpx afdocs check https://docs.example.com See afdocs.dev for full documentation, or install from npm. Library usage and CI integration are covered in the GitHub repo.\nBackground #This project grew out of findings from two research articles on agent documentation access patterns:\nAgent-Friendly Docs - Observations from 10+ hours of validating 578 coding patterns with Claude, covering URL failure modes, llms.txt discovery, markdown benefits, and page truncation. Agent Web Fetch Spelunking - Deep dive into how Claude Code\u0026rsquo;s web fetch pipeline processes HTML and markdown, including the summarization model, truncation limits, and why inline CSS can make a 97-line HTML page invisible to agents. Contributing #The specs are open for community review. We welcome feedback, proposed changes, platform data, and real-world results. See the GitHub repository for details.\nLicense #This work is licensed under CC BY 4.0.\n","date":null,"permalink":"https://agentdocsspec.com/","section":"Can agents read your documentation?","summary":"\u003cp\u003eDocumentation sites are increasingly consumed by coding agents (Claude Code,\nCursor, GitHub Copilot, and others) rather than human readers. But most sites\nweren\u0026rsquo;t built for this access pattern. Agents hit truncation limits, get walls\nof CSS instead of content, can\u0026rsquo;t follow cross-host redirects, and don\u0026rsquo;t know\nabout emerging discovery mechanisms like \u003ccode\u003ellms.txt\u003c/code\u003e.\u003c/p\u003e\n\u003cp\u003eThe result: agents frequently fail to get the documentation they need, fall\nback on training data, or work with partial information without knowing it.\u003c/p\u003e","title":"Can agents read your documentation?"},{"content":"Agent-friendly documentation has more than one surface. This page is the home for the specification family; each spec covers one surface and versions independently.\nAvailable now # Web Documentation Delivery Spec: 28 checks across 7 categories evaluating whether coding agents can discover, fetch, and receive a documentation site\u0026rsquo;s content intact. Covers llms.txt discovery, markdown availability, page size and truncation risk, content structure, URL stability, observability, and authentication/access. This spec was formerly served in full at this URL; it now lives at /spec/web/, split into per-category pages so that each page fits comfortably within agent fetch limits (its own recommendation, applied to itself). Planned #Companion specifications will be added here as the evidence base for them matures (see Related Surfaces in the web delivery spec):\nContent composition: what documentation should contain to serve agents well: factual consistency across pages, structure and density suited to machine consumption. Repository-local documentation: docs serving coding agents that work inside a repository, where discovery happens through grep and file listings rather than HTTP. Implementation #afdocs is the reference implementation of the web delivery spec: npx afdocs check \u0026lt;url\u0026gt; tests a documentation site against it.\n","date":null,"permalink":"https://agentdocsspec.com/spec/","section":"Specifications","summary":"\u003cp\u003eAgent-friendly documentation has more than one surface. This page is the\nhome for the specification family; each spec covers one surface and\nversions independently.\u003c/p\u003e\n\u003ch2 id=\"available-now\" class=\"relative group\"\u003eAvailable now \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#available-now\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h2\u003e\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/\" target=\"_blank\" rel=\"noreferrer\"\u003eWeb Documentation Delivery Spec\u003c/a\u003e\u003c/strong\u003e:\n28 checks across 7 categories evaluating whether coding agents can\ndiscover, fetch, and receive a documentation site\u0026rsquo;s content intact.\nCovers llms.txt discovery, markdown availability, page size and\ntruncation risk, content structure, URL stability, observability, and\nauthentication/access. This spec was formerly served in full at this\nURL; it now lives at \u003ccode\u003e/spec/web/\u003c/code\u003e, split into per-category pages so that\neach page fits comfortably within agent fetch limits (its own\nrecommendation, applied to itself).\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"planned\" class=\"relative group\"\u003ePlanned \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#planned\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h2\u003e\u003cp\u003eCompanion specifications will be added here as the evidence base for them\nmatures (see \u003ca href=\"https://agentdocsspec.com/spec/web/#related-surfaces\" target=\"_blank\" rel=\"noreferrer\"\u003eRelated Surfaces\u003c/a\u003e\nin the web delivery spec):\u003c/p\u003e","title":"Specifications"},{"content":" Status Draft Version 0.6.0 Date 2026-09-13 Author Dachary Carey + community contributors URL https://agentdocsspec.com Repository https://github.com/agent-ecosystem/agent-docs-spec Reference Implementation afdocs · npm · GitHub Spec Contents # Category 1: Content Discoverability: Checks for llms.txt existence, validity, size, link quality, and directives that point agents to it. Category 2: Markdown Availability: Checks for .md URL variants and content negotiation via Accept headers. Category 3: Page Size and Truncation Risk: Checks for rendering strategy, markdown/HTML/transfer size, content start position, and single-fetch completeness. Category 4: Content Structure: Checks for tabbed content serialization, header quality, code fence validity, link portability, and embedded data. Category 5: URL Stability and Redirects: Checks for HTTP status code honesty and redirect behavior. Category 6: Observability and Content Health: Checks for llms.txt coverage, markdown/HTML content parity, and cache header hygiene. Category 7: Authentication and Access: Checks for auth gating, alternative access paths, and bot-protection interference. Interaction Effects: How individual check failures combine into non-linear agent experience degradation. Appendices: Known platform truncation limits and notable exclusions. Changelog: Version history of the Agent-Friendly Documentation Spec. Abstract #Documentation sites are increasingly consumed by coding agents rather than human readers, but most sites are not built for this access pattern. Agents hit truncation limits, get walls of CSS instead of content, can\u0026rsquo;t follow cross-host redirects, and don\u0026rsquo;t know about emerging discovery mechanisms like llms.txt. This spec defines 28 checks across 7 categories that evaluate how well a documentation site serves agent consumers. It is grounded in empirical observation of real agent workflows and is intended as a shared standard for documentation teams, tool builders, and platform providers.\nScope #This spec targets coding agents that fetch documentation during real-time development workflows. These are tools like Claude Code, Cursor, GitHub Copilot, and similar IDE-integrated or CLI-based agents that a developer uses while writing code. The agent fetches a docs page, extracts information, and uses it to complete a task, all in a single session.\nThis spec does not target:\nTraining crawlers (GPTBot, ClaudeBot, etc.) that scrape content for model training. These have different access patterns, different user-agents, and different concerns. See Appendix B. Answer engines (Perplexity, Google AI Overviews, ChatGPT search) that retrieve content to generate responses to user queries. These systems have their own retrieval pipelines that may or may not resemble the web fetch pipelines described here. RAG pipelines that pre-index documentation into vector stores. These retrieve at query time from an index built ahead of time, so truncation limits and real-time fetch behavior are less relevant. Their ingestion step is a different matter: it fetches from the site like any automated client, and many of this spec\u0026rsquo;s checks serve it directly. See Serving RAG Ingestion Pipelines. The findings and checks in this spec are grounded in empirical observation of coding agents. Some recommendations (like providing llms.txt and serving markdown) will benefit other consumers too, but the pass/warn/fail criteria are calibrated for the coding agent use case.\nRelated Surfaces #This spec covers the delivery of web-served documentation: whether agents can discover, fetch, and receive content intact. Agent-friendly documentation has at least two other surfaces, deliberately out of scope here and planned as companion specifications as the evidence base for them matures:\nContent composition: what documentation should contain to serve agents well. Factual consistency across pages, structure and density suited to machine consumption, and how representation across a corpus shapes what agents effectively know about a product. These properties require semantic evaluation rather than the mechanical verification this spec\u0026rsquo;s checks are built on. Repository-local documentation: docs serving a coding agent that works inside a repository (README, docs/ directories, agent-facing instruction files). The agent discovers content with local tools (grep, find, directory listings) and consumes it through file reads, so almost none of the web spec\u0026rsquo;s checks transfer: there is no fetch pipeline, no truncation-by-summarizer, and no llms.txt; greppability and file layout do the work that discovery checks do on the web. Keeping these separate protects what makes each credible: this spec\u0026rsquo;s checks are mechanically verifiable against a live site, and that property should not be diluted as adjacent guidance develops.\nBackground #Agents don\u0026rsquo;t use docs like humans. They retrieve URLs from training data rather than navigating table-of-contents structures. They struggle with HTML-heavy pages, lose content to truncation with no indication anything was cut, and don\u0026rsquo;t know about emerging standards like llms.txt unless explicitly told. These checks codify the patterns that empirically help or hinder agent access to documentation content.\nTerminology # Agent: An LLM operating in an agentic coding workflow (e.g., Claude Code, Cursor, Copilot) that fetches and consumes documentation as part of a development task. See Scope for what this spec does and does not cover. Web fetch pipeline: The chain of processing between \u0026ldquo;agent requests a URL\u0026rdquo; and \u0026ldquo;model sees content.\u0026rdquo; Typically involves HTTP fetch, HTML-to-markdown conversion, truncation, and sometimes a summarization model. Trusted site: A domain hardcoded into an agent platform\u0026rsquo;s web fetch implementation that receives more favorable processing (e.g., bypassing summarization). Truncation: The silent removal of content that exceeds a platform\u0026rsquo;s size limit. The agent receives partial content with no indication that anything was cut. See Appendix A for known limits by platform. Conventions #This spec uses the following language to distinguish between requirements and recommendations:\nMust / Required: The item is an absolute requirement of the spec. Used sparingly; most checks in this spec are recommendations rather than hard requirements, because agent-friendliness is a spectrum. Should / Recommended: The item is a strong recommendation. There may be valid reasons to deviate, but the implications should be understood. May / Optional: The item is genuinely optional. Implementing it provides additional benefit but omitting it is not a deficiency. Sections of this spec are either normative (defining checks and their pass/warn/fail criteria) or informational (providing context, evidence, and recommendations). The distinction is noted where it matters:\nNormative sections: Category 1-7 check definitions, Checks Summary table. Informational sections: Background, Scope (including Related Surfaces), Start Here, \u0026ldquo;How Agents Get Content\u0026rdquo;, \u0026ldquo;Who Actually Uses llms.txt?\u0026rdquo;, Progressive Disclosure recommendation, \u0026ldquo;Making Private Docs Agent-Accessible\u0026rdquo;, Interaction Effects, \u0026ldquo;Serving RAG Ingestion Pipelines\u0026rdquo;, Appendices. Interaction Effects guides how implementations present combined results; it defines no pass/warn/fail criteria of its own. The progressive disclosure pattern for llms.txt is a recommendation from this spec, not a normative requirement. Sites that keep their llms.txt under 50,000 characters don\u0026rsquo;t need it.\nStart Here: Top Recommendations #If you\u0026rsquo;re a documentarian and can only do a few things, start with these. They are ordered by impact based on observed agent behavior:\nCreate an llms.txt that fits in a single agent fetch (under 50K characters). This is the single highest-impact action. Agents that find an llms.txt navigate documentation dramatically better. If your docs set is large, use the nested pattern to keep each file under the limit. Checks: llms-txt-exists, llms-txt-size\nServe markdown versions of your pages, and verify what you serve. Either via .md URL variants or content negotiation. Markdown is what agents actually want; HTML conversion is lossy and unpredictable. Then treat the markdown generator as a second rendering pipeline that needs its own QA: it can emit broken links or partial content while the HTML looks fine, and no human reads the markdown to notice. Checks: markdown-url-support, content-negotiation, markdown-link-portability, single-fetch-completeness\nKeep pages under 50,000 characters of content. If a page has tabbed or dropdown content, or generated data tables, that serialize into a massive blob, break it into separate pages or ensure the markdown version stays under the limit. Checks: page-size-markdown, page-size-html, tabbed-content-serialization, embedded-data-serialization\nPut a pointer to your llms.txt at the top of every docs page. A simple blockquote directive that tells agents where to find the documentation index. Anthropic does this; it works. Include it in both the HTML DOM and in markdown versions of pages. Checks: llms-txt-directive-html, llms-txt-directive-md\nDon\u0026rsquo;t break your URLs. If you must move content, use same-host HTTP redirects. Avoid cross-host redirects, JavaScript redirects, and soft 404s. Checks: http-status-codes, redirect-behavior\nMake sure bot protection isn\u0026rsquo;t blocking agents. Bot management tuned for scraper traffic can challenge, throttle, or stall automated clients fetching your public docs, and the failure is invisible from a browser. Exempt documentation routes from behavioral enforcement, or scope enforcement to interactive product surfaces. For affected sites this outranks everything else on this list: nothing above matters if agents can\u0026rsquo;t fetch at all. Checks: bot-protection-interference\nMonitor your agent-facing resources. Treat llms.txt and markdown endpoints like any other production surface: check freshness, verify content parity with HTML, and ensure cache headers allow timely updates. Checks: llms-txt-coverage, markdown-content-parity, cache-header-hygiene\nSpec Structure #Each check has:\nID: A short identifier (e.g., llms-txt-exists). Category: The area of agent-friendliness it evaluates. What it checks: A description of what the check evaluates. Why it matters: The observed agent behavior that motivates the check. Result levels: What constitutes a pass, warn, or fail. Recommended action: What to do to resolve a warn or failure state. Automation: Whether the check can be fully automated, partially automated (heuristic), or is advisory only. The canonical source of this spec is a single document (SPEC.md in the repository). The website serves it as per-category pages under agentdocsspec.com/spec/web/, following this spec\u0026rsquo;s own progressive disclosure recommendation: the full document outgrew the 100,000-character truncation threshold its own checks are built around, so it now practices what it prescribes. Each page stays well under the 50,000-character pass threshold.\nCheck Dependencies #Some checks depend on the results of others:\nllms-txt-valid, llms-txt-size, llms-txt-links-resolve, and llms-txt-links-markdown only run if llms-txt-exists passes. page-size-markdown only runs if markdown-url-support or content-negotiation passes (the site must serve markdown for this check to apply). page-size-html and content-start-position results should be flagged as unreliable if rendering-strategy fails (the measurements reflect a shell, not actual content). section-header-quality is most relevant when tabbed-content-serialization detects tabbed content. markdown-code-fence-validity only runs if markdown-url-support or content-negotiation passes (the site must serve markdown for this check to apply). It also runs against any discovered llms.txt files. llms-txt-coverage only runs if llms-txt-exists passes. auth-alternative-access only runs if auth-gate-detection returns warn or fail (the site must have auth-gated content for alternative access paths to be relevant). bot-protection-interference has no prerequisites, but it inverts the usual dependency direction: when it returns warn or fail, results from every multi-page check should be flagged as computed from a partial sample (see the Bot Protection Degrading Scan Reliability interaction effect). It is also evaluated from evidence gathered across the entire scan rather than run as a discrete Category 7 step. markdown-content-parity only runs if markdown-url-support or content-negotiation passes (the site must serve markdown for this check to apply). single-fetch-completeness and markdown-link-portability only run if the site serves markdown by some detected path: markdown-url-support or content-negotiation passes, or llms-txt-links-markdown finds markdown links. A site can serve agent-facing markdown through llms.txt alone, without page-level .md variants, and that markdown deserves the same evaluation. Implementations should run checks in category order (1 through 7) and skip dependent checks when their prerequisites fail.\nA Note on Responsible Use #This spec describes checks that involve making HTTP requests to documentation sites. Implementations should be respectful of the sites being evaluated: introduce delays between requests, cap concurrent connections, honor Retry-After headers, and avoid overwhelming sites with traffic. The goal is to help documentation teams improve agent accessibility, not to load-test their infrastructure.\nChecks Summary # ID Category Automation Severity Depends On llms-txt-exists Content Discoverability Full High \u0026ndash; llms-txt-valid Content Discoverability Full Medium llms-txt-exists llms-txt-size Content Discoverability Full High llms-txt-exists llms-txt-links-resolve Content Discoverability Full High llms-txt-exists llms-txt-links-markdown Content Discoverability Full Medium llms-txt-exists markdown-url-support Markdown Availability Full High \u0026ndash; content-negotiation Markdown Availability Full Medium \u0026ndash; rendering-strategy Page Size Heuristic High \u0026ndash; page-size-markdown Page Size Full High markdown-url-support or content-negotiation page-size-html Page Size Full High \u0026ndash; page-size-transfer Page Size Full Medium \u0026ndash; content-start-position Page Size Heuristic High \u0026ndash; single-fetch-completeness Page Size Heuristic Medium markdown-url-support, content-negotiation, or llms-txt-links-markdown tabbed-content-serialization Content Structure Heuristic High \u0026ndash; section-header-quality Content Structure Heuristic Medium tabbed-content-serialization markdown-code-fence-validity Content Structure Full Medium markdown-url-support or content-negotiation markdown-link-portability Content Structure Full Medium markdown-url-support, content-negotiation, or llms-txt-links-markdown embedded-data-serialization Content Structure Heuristic Medium \u0026ndash; http-status-codes URL Stability Full Medium \u0026ndash; redirect-behavior URL Stability Partial Medium \u0026ndash; llms-txt-directive-html Content Discoverability Heuristic High \u0026ndash; llms-txt-directive-md Content Discoverability Heuristic Medium markdown-url-support or content-negotiation llms-txt-coverage Observability Heuristic High llms-txt-exists markdown-content-parity Observability Heuristic Medium markdown-url-support or content-negotiation cache-header-hygiene Observability Full Medium \u0026ndash; auth-gate-detection Authentication Full High \u0026ndash; auth-alternative-access Authentication Partial Medium auth-gate-detection (warn or fail) bot-protection-interference Authentication Heuristic High \u0026ndash; Serving RAG Ingestion Pipelines #This section offers non-normative guidance for sites whose documentation is consumed by retrieval-augmented generation systems: documentation Q\u0026amp;A bots, docs MCP servers, and internal knowledge bases that chunk and embed content into a vector store. RAG retrieval happens at query time from that index, but ingestion has to get the content from somewhere first, and there are two common paths:\nWeb ingestion crawls the published site. It fetches like any automated client, so the properties that make a site agent-friendly at fetch time also make it index-friendly at ingestion time. The check mapping below applies to this path. Source ingestion reads the documentation source files (the markdown or MDX in the docs repository) directly, bypassing delivery entirely. This is common for teams indexing their own docs, and it is often a workaround for a site that is hostile to crawling; a site that passes this spec\u0026rsquo;s checks makes web ingestion a viable alternative. Source ingestion trades delivery problems for build-system problems: the ingester is effectively a second renderer, and build-time constructs (component tags, includes, frontmatter, variable substitution) reach the index unrendered unless the pipeline handles them the way the site generator does. The result can be an index of content nobody publishes. These are source-format processability concerns, out of scope for this spec; note that they are also distinct from the repository-local documentation surface in Related Surfaces, which concerns agents working interactively inside a repository, not batch pipelines reading its files. This guidance is informed by consumer reports from production RAG builds.\nHow existing checks serve web ingestion:\nCrawl manifest: llms-txt-exists and llms-txt-coverage give ingesters a complete, curated URL set instead of sitemap heuristics, and the descriptions provide per-page metadata for free. Clean source format: markdown-url-support and content-negotiation let pipelines ingest markdown directly and skip HTML extraction, which is the largest source of ingestion noise. Sites failing rendering-strategy or content-start-position poison their own index: boilerplate and empty shells get embedded and then retrieved. Chunk boundaries: section-header-quality matters doubly for RAG. Chunkers split on section boundaries, and a chunk\u0026rsquo;s header may be the only context it carries into retrieval. Self-describing headers produce self-describing chunks; generic headers (\u0026ldquo;Step 1\u0026rdquo;) produce chunks that retrieve poorly and confuse whatever reads them. Content that flows across section boundaries without markers chunks badly in practice. Incremental re-indexing: cache-header-hygiene is more valuable for RAG than for live fetch. ETag and Last-Modified let pipelines detect what changed and re-embed only that, instead of re-crawling wholesale or serving a stale index. Complete, navigable content: single-fetch-completeness and markdown-link-portability failures propagate undetected into an index. A paginated catalog ingests as a partial catalog; broken generated links embed as broken references. What this spec does not cover: embedding-friendly prose density and guidance on writing self-contained sections are content-composition concerns; see Related Surfaces. Content-categorization metadata (typed fields describing what a page is and what domain it serves) is a different kind of open question, under discussion in this project\u0026rsquo;s issue tracker: community vocabulary proposals exist and some platforms already serve frontmatter in markdown variants, but whether live agents benefit from receiving that metadata, or merely spend truncation budget on it, is unproven in either direction. This spec takes no position until there is evidence.\nContributing #This spec is a living document. Feedback, corrections, and contributions are welcome.\nDiscussion and feedback: Open an issue on the GitHub repository. Proposing changes: Submit a pull request. For significant changes (new checks, changes to pass/warn/fail criteria, new categories), please open an issue first to discuss the proposal. Platform truncation data: If you have data about a platform\u0026rsquo;s web fetch truncation limits (from official documentation, reverse engineering, or empirical testing), please contribute it to the Known Platform Limits table via issue or PR. Real-world validation: If you\u0026rsquo;ve run these checks against your own documentation site and have findings to share, we\u0026rsquo;d love to hear about it. References # llmstxt.org proposal Dachary Carey, \u0026ldquo;Agent-Friendly Docs\u0026rdquo; Dachary Carey, \u0026ldquo;Agent Web Fetch Spelunking\u0026rdquo; Giuseppe Gurgone, reverse-engineered Claude Code Web Fetch Mikhail Shilkov, \u0026ldquo;Claude Code Web Tools\u0026rdquo; Liran Yoffe, \u0026ldquo;Reverse Engineering Claude Code Web Tools\u0026rdquo; Checkly, \u0026ldquo;State of AI Agent Content Negotiation\u0026rdquo; Kody Jackson, \u0026ldquo;The speculative and soon to be outdated AI consumability scorecard\u0026rdquo; Longato, \u0026ldquo;LLMs.txt - Why Almost Every AI Crawler Ignores it\u0026rdquo; OtterlyAI, \u0026ldquo;llms.txt and AI Visibility: Results from OtterlyAI\u0026rsquo;s GEO Study\u0026rdquo; ","date":null,"permalink":"https://agentdocsspec.com/spec/web/","section":"Specifications","summary":"\u003ctable\u003e\n  \u003cthead\u003e\n      \u003ctr\u003e\n          \u003cth\u003e\u003c/th\u003e\n          \u003cth\u003e\u003c/th\u003e\n      \u003c/tr\u003e\n  \u003c/thead\u003e\n  \u003ctbody\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eStatus\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eDraft\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eVersion\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e0.6.0\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eDate\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e2026-09-13\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eAuthor\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003eDachary Carey + community contributors\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eURL\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"https://agentdocsspec.com\" target=\"_blank\" rel=\"noreferrer\"\u003ehttps://agentdocsspec.com\u003c/a\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eRepository\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"https://github.com/agent-ecosystem/agent-docs-spec\" target=\"_blank\" rel=\"noreferrer\"\u003ehttps://github.com/agent-ecosystem/agent-docs-spec\u003c/a\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n      \u003ctr\u003e\n          \u003ctd\u003e\u003cstrong\u003eReference Implementation\u003c/strong\u003e\u003c/td\u003e\n          \u003ctd\u003e\u003ca href=\"https://afdocs.dev\" target=\"_blank\" rel=\"noreferrer\"\u003e\u003ccode\u003eafdocs\u003c/code\u003e\u003c/a\u003e · \u003ca href=\"https://www.npmjs.com/package/afdocs\" target=\"_blank\" rel=\"noreferrer\"\u003enpm\u003c/a\u003e · \u003ca href=\"https://github.com/agent-ecosystem/afdocs\" target=\"_blank\" rel=\"noreferrer\"\u003eGitHub\u003c/a\u003e\u003c/td\u003e\n      \u003c/tr\u003e\n  \u003c/tbody\u003e\n\u003c/table\u003e\n\u003ch2 id=\"spec-contents\" class=\"relative group\"\u003eSpec Contents \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#spec-contents\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h2\u003e\u003cul\u003e\n\u003cli\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/content-discoverability/\" target=\"_blank\" rel=\"noreferrer\"\u003eCategory 1: Content Discoverability\u003c/a\u003e: Checks for llms.txt existence, validity, size, link quality, and directives that point agents to it.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/markdown-availability/\" target=\"_blank\" rel=\"noreferrer\"\u003eCategory 2: Markdown Availability\u003c/a\u003e: Checks for .md URL variants and content negotiation via Accept headers.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/page-size/\" target=\"_blank\" rel=\"noreferrer\"\u003eCategory 3: Page Size and Truncation Risk\u003c/a\u003e: Checks for rendering strategy, markdown/HTML/transfer size, content start position, and single-fetch completeness.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/content-structure/\" target=\"_blank\" rel=\"noreferrer\"\u003eCategory 4: Content Structure\u003c/a\u003e: Checks for tabbed content serialization, header quality, code fence validity, link portability, and embedded data.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/url-stability/\" target=\"_blank\" rel=\"noreferrer\"\u003eCategory 5: URL Stability and Redirects\u003c/a\u003e: Checks for HTTP status code honesty and redirect behavior.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/observability/\" target=\"_blank\" rel=\"noreferrer\"\u003eCategory 6: Observability and Content Health\u003c/a\u003e: Checks for llms.txt coverage, markdown/HTML content parity, and cache header hygiene.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/authentication/\" target=\"_blank\" rel=\"noreferrer\"\u003eCategory 7: Authentication and Access\u003c/a\u003e: Checks for auth gating, alternative access paths, and bot-protection interference.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/interaction-effects/\" target=\"_blank\" rel=\"noreferrer\"\u003eInteraction Effects\u003c/a\u003e: How individual check failures combine into non-linear agent experience degradation.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/appendices/\" target=\"_blank\" rel=\"noreferrer\"\u003eAppendices\u003c/a\u003e: Known platform truncation limits and notable exclusions.\u003c/li\u003e\n\u003cli\u003e\u003ca href=\"https://agentdocsspec.com/spec/web/changelog/\" target=\"_blank\" rel=\"noreferrer\"\u003eChangelog\u003c/a\u003e: Version history of the Agent-Friendly Documentation Spec.\u003c/li\u003e\n\u003c/ul\u003e\n\u003ch2 id=\"abstract\" class=\"relative group\"\u003eAbstract \u003cspan class=\"absolute top-0 w-6 transition-opacity opacity-0 -start-6 not-prose group-hover:opacity-100\"\u003e\u003ca class=\"group-hover:text-primary-300 dark:group-hover:text-neutral-700\" style=\"text-decoration-line: none !important;\" href=\"#abstract\" aria-label=\"Anchor\"\u003e#\u003c/a\u003e\u003c/span\u003e\u003c/h2\u003e\u003cp\u003eDocumentation sites are increasingly consumed by coding agents rather than\nhuman readers, but most sites are not built for this access pattern. Agents\nhit truncation limits, get walls of CSS instead of content, can\u0026rsquo;t follow\ncross-host redirects, and don\u0026rsquo;t know about emerging discovery mechanisms like\n\u003ccode\u003ellms.txt\u003c/code\u003e. This spec defines 28 checks across 7 categories that evaluate how\nwell a documentation site serves agent consumers. It is grounded in empirical\nobservation of real agent workflows and is intended as a shared standard for\ndocumentation teams, tool builders, and platform providers.\u003c/p\u003e","title":"Web Documentation Delivery Spec"}]