We are excited to roll out our new llms.txt architecture to all Mintlify powered docs. On large docs sites, agents navigate 52% faster while using 45% fewer tokens.
The llms.txt file was introduced in 2024 to provide agents with a navigation map of a website’s content.
However, if an llms.txt file gets too large, agents will receive large amounts of irrelevant information that bloats their context window, leading to more token spend and slower task completions. When we added /llms.txt after its initial proposal in 2024, we dealt with this by adding a 100,000 character cut off. But over the past two years, as the docs sites we power have grown, the limits of this temporary solution have started to be much more noticeable.
To preserve coverage without expanding the root indefinitely, we rebuilt Mintlify's llms.txt generation architecture as a hierarchy of files, where the generator recursively identifies and moves complete navigation sections into child files through docs.json.
On 200 tasks across ten docs sites, our improved version of llms.txt enabled agents to complete tasks 52.1% faster (363.6s to 174.2s), with 44.8% less tokens (1.74m to 960k).
Find the full blog below: