↓Skip to main content

Can agents read your documentation?

For AI agents: a documentation index is available at /llms.txt — markdown versions of all pages are available by appending index.md to any URL path.

Documentation sites are increasingly consumed by coding agents (Claude Code, Cursor, GitHub Copilot, and others) rather than human readers. But most sites weren’t built for this access pattern. Agents hit truncation limits, get walls of CSS instead of content, can’t follow cross-host redirects, and don’t know about emerging discovery mechanisms like llms.txt.

The result: agents frequently fail to get the documentation they need, fall back on training data, or work with partial information without knowing it.

The Web Documentation Delivery Spec defines 28 checks across 7 categories that evaluate how well a documentation site serves agent consumers.

Category Checks What it evaluates
Content Discoverability 7 Discovery index exists, is valid, fits in a single fetch, links resolve, links point to markdown, embedded directives in HTML and markdown pointing agents to llms.txt
Markdown Availability 2 .md URL support, content negotiation via Accept headers
Page Size 6 Rendering strategy (SPA/CSR detection), markdown size, HTML size (pre/post conversion), served transfer size, content start position, single-fetch completeness
Content Structure 5 Tabbed content serialization blowup, section header quality, code fence validity, markdown link portability, embedded data serialization
URL Stability 2 Soft 404 detection, redirect behavior
Observability 3 llms.txt coverage, markdown/HTML content parity, cache header hygiene
Authentication 3 Auth gate detection, alternative access paths for gated content, bot-protection interference with automated fetching

Each check has defined pass/warn/fail criteria, an automation level, and severity.

Read the Spec

Web delivery is the first surface in a planned family of specifications. Companion specs for content composition (what documentation should contain to serve agents well) and repository-local documentation (docs agents grep and read inside codebases) will land at Specifications as the evidence base for them matures.

For empirical observations on how specific agent platforms (Claude, Cursor, Copilot, Gemini, and others) handle retrieval, truncation, and summarization, see Agent platform comparisons.

Quick Start for Documentarians #

If you can only do a few things, these have the highest impact:

  1. Create an llms.txt under 50K characters. This is the single most effective discovery mechanism observed.
  2. Serve markdown versions of your pages via .md URLs or content negotiation, and verify what you serve: generated markdown can ship broken links or partial content while the HTML looks fine.
  3. Keep pages under 50K characters of content. Break up mega-pages, serialized tab content, and generated data tables.
  4. Add an llms.txt pointer to the top of every docs page.
  5. Don’t break your URLs. Use same-host HTTP redirects if you must move content.
  6. Make sure bot protection isn’t blocking agents. Exempt public docs routes from behavioral enforcement; for affected sites, this outranks everything else here.
  7. Monitor your agent-facing resources. Keep llms.txt fresh, verify markdown parity, and check cache headers.

Test Your Docs #

afdocs is a companion CLI tool and Node.js library that implements this spec. It runs the automated checks against your documentation site and reports what’s working, what’s not, and what to fix.

npx afdocs check https://docs.example.com

See afdocs.dev for full documentation, or install from npm. Library usage and CI integration are covered in the GitHub repo.

Background #

This project grew out of findings from two research articles on agent documentation access patterns:

  • Agent-Friendly Docs - Observations from 10+ hours of validating 578 coding patterns with Claude, covering URL failure modes, llms.txt discovery, markdown benefits, and page truncation.
  • Agent Web Fetch Spelunking - Deep dive into how Claude Code’s web fetch pipeline processes HTML and markdown, including the summarization model, truncation limits, and why inline CSS can make a 97-line HTML page invisible to agents.

Contributing #

The specs are open for community review. We welcome feedback, proposed changes, platform data, and real-world results. See the GitHub repository for details.

License #

This work is licensed under CC BY 4.0.