Can agents read your documentation? on Agent-Friendly Documentation Spec

Category 1: Content Discoverability

Mon, 01 Jan 0001 00:00:00 +0000

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether agents can find and navigate the site’s documentation content. This includes whether the site provides an llms.txt file, whether that file is useful to agents, and whether documentation pages include signals that direct agents to discovery resources.

Location Discovery #

The llmstxt.org proposal specifies that llms.txt should be at the root path (/llms.txt), mirroring robots.txt and sitemap.xml. In practice, the location varies significantly across sites:

Category 2: Markdown Availability

Mon, 01 Jan 0001 00:00:00 +0000

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether the site serves documentation in markdown format, which agents consume far more effectively than HTML.

`markdown-url-support` #

What it checks: Whether appending .md to documentation page URLs returns valid markdown content.
Why it matters: Agents work dramatically better with markdown than HTML. The HTML-to-markdown conversion in web fetch pipelines is lossy and unpredictable. Sites that serve markdown directly bypass conversion issues entirely. However, agents don’t discover this pattern on their own; it needs to be signaled.
Result levels:
- Pass: .md URLs return valid markdown with 200 status.
- Warn: Some pages support .md but not consistently.
- Fail: .md URLs return errors or HTML.
Recommended action:
- Warn: Ensure all documentation pages serve markdown when .md is appended to the URL, not just some.
- Fail: Configure your docs platform to serve .md variants for all documentation pages.
Automation: Full. Test against a sample of page URLs (from llms.txt, sitemap, or user-provided list).

`content-negotiation` #

What it checks: Whether the server responds to Accept: text/markdown with markdown content and an appropriate Content-Type header.
Why it matters: Some agents (Claude Code, Cursor, OpenCode) send Accept: text/markdown as their preferred content type. If the server honors this, the agent gets clean markdown without needing to know about .md URL patterns. Most agents don’t request markdown, but the ones that do should get it.
Result levels:
- Pass: Server returns markdown content with Content-Type: text/markdown when requested.
- Warn: Server returns markdown content but with incorrect Content-Type.
- Fail: Server ignores the Accept header and returns HTML regardless.
Recommended action:
- Warn: Set the response Content-Type to text/markdown when serving markdown content. The correct header enables optimizations in some agent pipelines.
- Fail: Configure your server to honor Accept: text/markdown requests and return markdown content. Some agents (Claude Code, Cursor, OpenCode) request markdown this way.
Automation: Full.

Category 3: Page Size and Truncation Risk

Mon, 01 Jan 0001 00:00:00 +0000

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether page content fits within the transfer and processing limits of agent web fetch pipelines. Truncation is silent: the agent doesn’t know it’s working with partial data.

How Agents Get Content #

Not all agents see the same thing. The format an agent receives depends on the request it makes and the server’s response:

Category 4: Content Structure

Mon, 01 Jan 0001 00:00:00 +0000

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether page content is structured in ways that agents can effectively consume. These are harder to fully automate and rely more on heuristics.

`tabbed-content-serialization` #

What it checks: Whether pages use tabbed, accordion, or dropdown UI patterns that serialize into long sequential content in the source, and if so, how large the serialized output is.
Why it matters: Tabbed content is great for humans but can be catastrophic for agents. A tutorial with 11 language variants serializes into a single massive document where an agent might see only the first 1-3 variants. Source order determines what the agent sees; everything past the truncation point is invisible. Asking for a specific variant (e.g., Python) does not help if that variant is beyond the truncation point.
Result levels:
- Pass: No tabbed content, or tabbed content that serializes to under 50,000 characters total.
- Warn: Tabbed content serializes to 50,000-100,000 characters.
- Fail: Tabbed content serializes to over 100,000 characters.
Recommended action: Break tab variants into separate pages, or provide a mechanism for agents to request specific variants. Agents see only the first few variants; content in later tabs is truncated.
Automation: Heuristic. Detect common tab/accordion component patterns (e.g., <Tab>, <Tabs>, role=“tabpanel”, common CSS class patterns) and estimate serialized size.

`section-header-quality` #

What it checks: Whether section headers contain enough context to be meaningful without the surrounding UI. Specifically, when tabbed content is serialized, do headers distinguish which variant (language, platform, deployment type) a section belongs to?
Why it matters: When an agent sees serialized tabbed content, descriptive headers are the only way it can tell which section applies to which context. Generic headers like “Step 1” repeated across all variants are indistinguishable. Headers like “Step 1 (Python/PyMongo)” preserve the filtering context that the UI provided to human readers.
Result levels (evaluated both within individual tab groups and across tab groups on the same page; the overall result is the worst of both):
- Pass: <=25% of headers within tabbed sections are generic (repeated across variants without distinguishing context).
- Warn: 25-50% of headers are generic across variants.
- Fail: >50% of headers are generic, or identical header sets are repeated across separate tab groups on the same page with no variant context. These thresholds are defaults; implementations should allow them to be configured.
Recommended action: Add variant context to headers (e.g., “Step 1 (Python)” instead of “Step 1”) so agents can distinguish which section belongs to which variant when content is serialized.
Automation: Heuristic. Requires detecting tabbed sections and analyzing header patterns within them.

`markdown-code-fence-validity` #

What it checks: Whether markdown content contains unclosed or improperly nested code fences (``` or ~~~ blocks without a matching closing delimiter).
Why it matters: An unclosed code fence causes everything after it to be interpreted as code rather than prose. The agent sees documentation text, API descriptions, and instructions as if they were inside a code block, which fundamentally changes how it processes the content. A model treats code blocks as literal content to reproduce or analyze, not as natural language instructions to follow. If an unclosed fence appears early in a page, the agent effectively loses the rest of the document’s meaning. This applies to any markdown the site serves directly: pages via .md URLs or content negotiation, and llms.txt files themselves.
Result levels:
- Pass: All code fences in the markdown content are properly opened and closed.
- Fail: One or more unclosed code fences detected.
Recommended action: Ensure every opening ``` or ~~~ has a matching closing delimiter. Everything after an unclosed fence is interpreted as code, causing agents to misread documentation as literal content.
Notes on delimiter matching: Per the CommonMark spec, a backtick fence (```) can only be closed by another backtick fence of equal or greater length, and likewise for tilde fences (~~~). Opening with ``` and attempting to close with ~~~ leaves the backtick fence unclosed. There is no intermediate “mismatched but balanced” state; mismatched delimiters produce unclosed fences and should be reported as failures.
Automation: Full. Parse the markdown for fence delimiters (``` and ~~~, with optional info strings) and verify each opening delimiter has a matching close. Run against markdown served via .md URLs, content negotiation responses, and llms.txt files.
Notes: This check applies to markdown the site authors and serves directly. Code fences broken by an HTML-to-markdown conversion pipeline are outside the site owner’s control, though implementations may optionally flag them as informational findings.

`markdown-link-portability` #

What it checks: Whether links in served markdown are absolute URLs, and whether a sample of them resolves to the representation they promise (a .md link returns markdown content, not an HTML error page).

Category 5: URL Stability and Redirects

Mon, 01 Jan 0001 00:00:00 +0000

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether documentation URLs behave in ways that agents can handle, given that agents retrieve URLs from training data and have limited ability to discover moved content.

`http-status-codes` #

What it checks: Whether pages return correct HTTP status codes. In particular, whether “not found” pages return 404 (not 200 with a friendly error page).
Why it matters: Soft 404s (200 status with “page not found” content) are worse than real 404s for agents. The agent sees a 200 and tries to extract information from the error page content rather than recognizing the page doesn’t exist. A clean 404 tells the agent to try a different approach.
Result levels:
- Pass: Error pages return appropriate 4xx status codes.
- Fail: Error pages return 200 (soft 404).
Recommended action: Configure your server to return 404 status codes for pages that don’t exist. Agents try to extract information from soft 404 page content instead of recognizing the page is missing.
Automation: Full. Test known-bad URLs (e.g., append random strings to real page paths) and check status codes.

`redirect-behavior` #

What it checks: Whether redirects are same-host (transparent to agents) or cross-host (a friction point), and whether redirects use proper HTTP status codes (301/302) vs. JavaScript-based redirects.
Why it matters: Same-host redirects work transparently because the HTTP client follows them automatically. Cross-host redirects are a known failure point; Claude Code, for example, doesn’t automatically follow cross-host redirects (security measure against open-redirect attacks). JavaScript redirects don’t work at all because agents don’t execute JavaScript.
Result levels:
- Pass: All redirects are same-host HTTP redirects (301/302).
- Warn: Cross-host HTTP redirects are present (agents may or may not follow them depending on the platform).
- Fail: JavaScript-based redirects are detected.
Recommended action:
- Warn: Where possible, use same-host redirects or update URLs to point directly to the final destination.
- Fail: Replace JavaScript-based redirects with HTTP 301/302 redirects. Agents don’t execute JavaScript and will not follow these redirects.
Automation: Partial. HTTP redirects are detectable. JavaScript redirects require fetching the page and scanning for window.location, meta refresh, or similar patterns.

Category 6: Observability and Content Health

Mon, 01 Jan 0001 00:00:00 +0000

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether the site’s agent-facing resources stay accurate and up to date over time. Categories 1-5 can be evaluated as point-in-time audits; this category addresses the ongoing maintenance dimension. llms.txt files and markdown endpoints are secondary outputs that often aren’t wired into existing monitoring, so they can go stale, break, or drift from primary HTML content without anyone noticing.

Category 7: Authentication and Access

Mon, 01 Jan 0001 00:00:00 +0000

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

These checks evaluate whether documentation is accessible to agents at all: without requiring interactive authentication, and without infrastructure-level barriers aimed at automated clients. Docs behind login walls are effectively invisible to coding agents, which has significant implications as agent-assisted development becomes a standard workflow. Bot-protection systems can produce the same invisibility through a different mechanism, and often without the site owner realizing documentation is affected.

Interaction Effects

Mon, 01 Jan 0001 00:00:00 +0000

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

Individual checks measure discrete properties, but agent experience can degrade non-linearly when certain failures combine. A site might pass most checks individually while still being effectively inaccessible to agents because of how the failures interact. This section describes known interaction patterns that implementations should detect and surface. Implementations should evaluate these after all individual checks have completed.

Appendices

Mon, 01 Jan 0001 00:00:00 +0000

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

Appendix A: Known Platform Truncation Limits #

The thresholds used in this spec’s pass/warn/fail levels are derived from observed and documented platform behavior. This appendix tracks known limits so that implementations can calibrate their thresholds appropriately, and so that the spec’s default thresholds can be updated as more data becomes available.

Changelog

Mon, 01 Jan 0001 00:00:00 +0000

Part of the Web Documentation Delivery Spec. The Checks Summary lists all checks with links to their definitions.

v0.6.0 (2026-09-13) #

Added bot-protection-interference (Category 7: Authentication and Access). Detects bot-protection systems interfering with automated documentation fetching: challenge interstitials served as 200, tarpits that return headers and then stall the response body indefinitely, and volume-triggered throttling or blocking. Grounded in an observed production case where CDN bot management responded to a sustained scan by holding response bodies open; single-request probes looked healthy throughout, and enforcement decayed after a cooldown. Detection is heuristic, observed as a byproduct of a normal scan rather than a directed probe.
Added the Bot Protection Degrading Scan Reliability interaction effect: when enforcement engages mid-scan, other checks score the surviving sample, so implementations should aggregate fetch failures at run level and flag results when the failure rate is high.
Expanded the Category 7 introduction to cover infrastructure-level access barriers alongside authentication.
Added page-size-transfer (Category 3: Page Size and Truncation Risk). Measures the served byte size of the HTML document response, a failure mode page-size-html’s post-conversion measurement cannot see. Grounded in measurements of production documentation sites where 75-84% of page bytes were serialized framework payloads (component trees, resolved metadata, embedded duplicate markdown source), with served-bytes-to-content ratios from 40:1 to 200:1. Scored on served bytes; the ratio is reported as a diagnostic.
Documented the pipeline distinction in Category 3’s “How Agents Get Content” section: convert-then-truncate, truncate-then-convert (or raw ingestion), and capped fetch, and which size measurement predicts each.
Added single-fetch-completeness (Category 3: Page Size and Truncation Risk). Detects pagination in markdown responses and verifies the continuation is machine-followable. Grounded in an observed production catalog whose markdown variant showed 100 of 102 entries with a trailing relative continuation URL that returned an empty body, while the complete set would have fit well under the 50,000-character pass threshold.
Added markdown-link-portability (Category 4: Content Structure). Links in served markdown should be absolute and should resolve to the representation they promise, verified beyond status codes (content type and soft-404 heuristics). Grounded in an observed production catalog whose generated markdown links all pointed into a wrong path prefix and soft-404ed as HTML SPA shells at .md URLs while returning 200.
Added embedded-data-serialization (Category 4: Content Structure). Attributes page size to machine-generated bulk elements (large uniform tables, inline data blobs). Grounded in a measured production reference page: 302KB of HTML, 64% table markup including a single 218-row generated table, converting to ~83,000 characters while its non-table prose totaled ~17,000.
Extended markdown-content-parity notes with guidance for dynamically generated pages: compare item counts between representations and distinguish default-filter divergence, pagination windowing, and staleness as causes.
Added the Dynamic Content Rendered Statically interaction effect: the four characteristic ways a dynamic page flattens badly into static markdown (too much, too little, inconsistent, unnavigable), observed together on a single production catalog page.
Added the Related Surfaces subsection to Scope, naming the planned companion specifications (content composition, repository-local documentation) and the boundary that keeps this spec’s checks mechanically verifiable.
Added the Serving RAG Ingestion Pipelines informational section, mapping existing checks to RAG ingestion needs (crawl manifest, clean source, chunk boundaries, incremental re-indexing), informed by consumer reports from production RAG builds. The Scope section’s RAG exclusion now distinguishes query-time retrieval (out of scope) from ingestion (served by this spec).
Restructured the website serving of the spec: the full document exceeded the 100,000-character truncation threshold its own checks warn about, so it is now served as per-category pages under /spec/web/, with /spec/ becoming a landing page for this and future companion specifications. The canonical source remains a single SPEC.md in the repository. This breaks previously published deep URLs deliberately, in exchange for a namespace that accommodates the companion specs.
Check count: 23 → 28.

v0.5.1 (2026-05-08) #

Moved per-platform truncation data out of Appendix A into a new Platforms comparison page on the site. Appendix A retains the spec’s threshold rationale and points readers to the platforms page for current per-platform observations. Category 3 (Page Size and Truncation Risk) now references the platforms page so readers can connect threshold choices to empirical pipeline behavior. No threshold or check definitions changed. Platforms page authored by Rhyannon Rodriguez.

v0.5.0 (2026-04-25) #

Split llms-txt-directive into two independent checks: llms-txt-directive-html and llms-txt-directive-md. The original check conflated two distinct signals that serve different audiences. The HTML check detects directives in the rendered DOM (for agents fetching HTML pages); the markdown check detects directives in markdown source (for agents fetching .md URLs or using content negotiation). The split also adds explicit detection guidance: incidental mentions of llms.txt in navigation, metadata, or page content discussing the feature do not count as directives. llms-txt-directive-md depends on markdown-url-support or content-negotiation; it is skipped if neither passes. This is a breaking change for implementations that reference the old check ID.
Check count: 22 → 23.

v0.4.0 (2026-04-21) #

Renamed llms-txt-freshness to llms-txt-coverage. The check compares llms.txt URLs against the sitemap to measure how much of the site is represented; that’s coverage, not freshness. Whether listed URLs still resolve is already handled by llms-txt-links-resolve. Rewrote the check description to match. This is a breaking change for implementations that reference the old check ID.
Revised page-size-html and content-start-position to be pipeline-agnostic. The previous language prescribed a specific conversion approach (Turndown with default configuration) based on one agent’s behavior. Agent HTML processing pipelines vary and continue to evolve; the spec now describes the measurement goal (approximate what agents see) and leaves conversion details to implementers. Recommended actions now cover all boilerplate sources (navigation, sidebars, serialized tabbed content) rather than focusing narrowly on inline CSS/JS.
Expanded llms-txt-coverage to account for intentional curation. Many sites deliberately include only a subset of pages in llms.txt (excluding changelogs, old versions, directory pages, etc.). The check now describes three use cases (full parity, curated, hybrid) served by configurable thresholds and exclusion patterns, rather than treating all gaps as problems.
Expanded markdown-content-parity to distinguish intentional audience segmentation from unintentional content drift. Some sites intentionally serve different content per audience (agent-optimized markdown vs. human-optimized HTML). The check now describes audience-segmentation tags as a mechanism implementations can recognize, and supports the same mirrored/segmented/curated spectrum as llms-txt-coverage. The spec does not prescribe specific tag conventions; implementations document which they support.

v0.3.0 (2026-03-31) #

Merged Category 6 (Agent Discoverability Directives) into Category 1, renamed to “Content Discoverability.” The llms-txt-directive check (now llms-txt-directive-html and llms-txt-directive-md) answers the same fundamental question as the llms.txt checks: can agents find and navigate the content? This reduces categories from 8 to 7.
Renumbered Category 7 (Observability) to 6, Category 8 (Authentication) to 7.
Added Recommended action field to all 22 check definitions. Provides 1-2 sentence actionable guidance for each warn and fail state, giving documentation teams a clear next step rather than just a diagnosis.
Added Interaction Effects section after Checks Summary. Documents six patterns where combinations of check results indicate systemic problems worse than individual failures suggest (e.g., undiscoverable markdown, no viable content path, oversized pages without markdown escape).
Category count: 8 → 7. Check count unchanged at 22.

v0.2.1 (2026-03-15) #

Clarifications from implementing the afdocs conformance tool against the spec. No new checks; all changes refine existing check definitions.

Can agents read your documentation? on Agent-Friendly Documentation Spec

Category 1: Content Discoverability

Location Discovery #

Category 2: Markdown Availability

markdown-url-support #

content-negotiation #

Category 3: Page Size and Truncation Risk

How Agents Get Content #

Category 4: Content Structure

tabbed-content-serialization #

section-header-quality #

markdown-code-fence-validity #

markdown-link-portability #

Category 5: URL Stability and Redirects

http-status-codes #

redirect-behavior #

Category 6: Observability and Content Health

Category 7: Authentication and Access

Interaction Effects

Appendices

Appendix A: Known Platform Truncation Limits #

Changelog

v0.6.0 (2026-09-13) #

v0.5.1 (2026-05-08) #

v0.5.0 (2026-04-25) #

v0.4.0 (2026-04-21) #

v0.3.0 (2026-03-31) #

v0.2.1 (2026-03-15) #

`markdown-url-support` #

`content-negotiation` #

`tabbed-content-serialization` #

`section-header-quality` #

`markdown-code-fence-validity` #

`markdown-link-portability` #

`http-status-codes` #

`redirect-behavior` #