Improve docs agent readiness

[joshuablum]

Our docs currently don't score that high on Cloudflare's https://isitagentready.com and there is a lot of room for improvement. We already serve pages as markdown when appending .md to a URL (basic right now and some undiscoverable) and also have a llms.txt file in place (misses a few things). Those can be improved as well as a lot of other changes and improvements (e.g. much more in-depth robots.txt file). This PR implements several recommendations and best practices from https://isitagentready.com, https://blog.cloudflare.com/agent-readiness/, and https://blog.cloudflare.com/aeo/.

Discoverability

• robots.txt is now a route instead of a two line static file. It has a Sitemap: directive (there wasn't one), Content Signals declaring search=yes, ai-input=yes, ai-train=yes, and explicit groups for 12 named AI crawlers.
• Removed a dead lookup from sitemap.xml. It opened by fetching and entry ID (6aa5449b...) which doesn't exist anymore.

Content accessibility

• llms.txt went from 128 to 613 entries. It only walked one level deep, so nested pages like /getting-started/installing/laravel were dropped and the ~460 reference entries (tags, modifiers, fieldtypes, variables) weren't in there at all. An agent reading it couldn't discover any of it.
• The home page now has a Markdown version at /index.md. None before.
• Every page links to its Markdown twin via <link rel="alternate" type="text/markdown">, plus rel="canonical" and a robots tag with max-snippet:-1 so answer engines can quote a full passage. /search-results is noindex.

Meta descriptions

• 225 of 630 entries had no intro or description, like the modifiers, variables, or
  resource API pages. They were emitting empty meta descriptions, empty Open Graph tags and empty llms.txt entries.
• There's now a fallback chain (intro -> description -> first paragraph of the body) that strips code fences, ::tabs, :::tip blocks and inline markdown. It's a computed
  value so one implementation covers all templates and also the llms.txt.

Two other bugs fixed

• Link anchors were being mangled in the Markdown output. E.g. (#update-scripts) became
  (#update-scripts.md). The .md now goes before the fragment.
• An unpublished draft was listed in llms.txt. digitalocean.md is
  published: false but was included in the file. Now correctly filter the entries.

[joshuablum]

Added some more things to improve the agent readiness:

• Properly redirect legacy URLs to their canonical twins to prevent 404s. Example: Request to /users.md now follows the existing HTML redirect rules and resolves to the destination's Markdown equivalent, such as /control-panel/users.md, instead of returning a 404.
• Middleware to properly handle text/markdown requests so docs pages support correct HTTP content negotiation. A request with Accept: text/markdown receives the markdown representation at the original URL.
• Added Link headers to docs responses so HTML responses advertise their markdown alternative as well as the /llms.txt. Also the other way around so markdown responses advertise the HTML page and /llms.txt.
• Added some missing redirects