Tracking issue (Rustdoc): hoedown -> pulldown migration

[nrc]

If you've arrived via a warning and what to know what is going on, see this comment.

• implement Pulldown rendering behind a flag (Replace hoedown with pull in rustdoc #40338, Re-enable hoedown by default #41431)
• implement warnings for changes (Add warnings when rustdoc html rendering differs #41991)
• begin warning cycle (Print rustdoc rendering warnings all the time #45324)
• move to Pulldown by default (Switch to pulldown as default markdown renderer #47398)
• remove Hoedown, the difference warnings, the html-diff dependency, etc (Remove hoedown from rustdoc #48274)

Blocking warning cycle

• move warnings to off by default (on if user chooses Pulldown rendering)
• format warning, and add information about how to remedy
• assess and mitigate false positives
• advertise warnings (https://blog.guillaume-gomez.fr/articles/2017-09-18+New+rustdoc+rendering+common+errors)
• move warnings to on by default

[nrc]

cc @rust-lang/dev-tools @rust-lang/docs

[nrc]

What is happening?

We're changing the way that Rustdoc renders markdown in your docs. We're moving to use a new parser and renderer (Pulldown) which is standards compliant (CommonMark), more robust, less buggy, and written in Rust. You can use Pulldown on nightly rustdoc by using the --enable-commonmark command line argument.

However, there are some changes to how your docs might be rendered. The known issues are:

• Pulldown does not support superscripts such as 2^5, you must use 2<sup>5</sup> instead.
• Pulldown identifies more examples as code blocks
• Pulldown identifies numbered lists in more places than Hoedown
• Pulldown requires a newline between a paragraph and any link reference definitions
• 'automatic links' in Pulldown need <> around them
• Hoedown required a newline between paragraphs and lists, Pulldown does not.
• A space is required after ** to work as emphasis in Pulldown
• Many insignificant changes to whitespace in generated HTML, etc.

We are currently providing warnings (when running Rustdoc) if the documentation for your crate will change with the new renderer. You should check these warnings to ensure that your documentation still renders correctly.

[nrc]

Update: #44238 and #44329 addressed some issues with the warnings and it is now off by default (it was on by default for one nightly release). The next step is to assess the false positive situation. @QuietMisdreavus is going to list some good test crates and obtain the warning output for them. @GuillaumeGomez is going to try some experiments to reduce the false positive rate.

[nrc]

ping @QuietMisdreavus did you get a chance to look at good test crates?

[QuietMisdreavus]

After getting the first false positive i didn't dig much farther. However, a quick check of a handful of popular crates shows some more leads:

time looks like it has some true positives, in the form of ^ not being <sup>:

Details

 Documenting time v0.1.38
WARNING: documentation for this crate may be rendered differently using the new Pulldown renderer.
    See https://github.com/rust-lang/rust/issues/44229 for details.
WARNING: rendering difference in `Simple time handling....`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/time-0.1.38/src/lib.rs:11:0
    /html[0]/body[1]/p[6] Text differs:
        expected: `... [strftime()]...`
        found:    `...`
    /html[0]/body[1]/p[6] Unexpected element `a`: found: `<a href="http://man7.org/linux/man-pages/man3/strf ... html">strftime()</a>`
    /html[0]/body[1]/p[6] Unexpected element: ` function from the C...`
WARNING: rendering difference in `Identifies the time zone that was used to compute this broken-down time...`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/time-0.1.38/src/lib.rs:336:4
    /html[0]/body[1]/p[0] Text differs:
        expected: `...`
        found:    `...*60*60 = -25200.`
    /html[0]/body[1]/p[0] One element is missing: expected: `em`
    /html[0]/body[1]/p[0] One element is missing: expected: `0`
WARNING: rendering difference in `Returns the total number of whole microseconds in the duration,...`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/time-0.1.38/src/duration.rs:197:4
    /html[0]/body[1]/p[0] Text differs:
        expected: `on overflow (exceeding 2^63 microseconds in either direction).`
        found:    `on overflow (exceeding 2`
    /html[0]/body[1]/p[0] Unexpected element `sup`: found: `<sup>63</sup>`
    /html[0]/body[1]/p[0] Unexpected element: ` microseconds in either direction).`
WARNING: rendering difference in `Returns the total number of whole nanoseconds in the duration,...`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/time-0.1.38/src/duration.rs:205:4
    /html[0]/body[1]/p[0] Text differs:
        expected: `on overflow (exceeding 2^63 nanoseconds in either direction).`
        found:    `on overflow (exceeding 2`
    /html[0]/body[1]/p[0] Unexpected element `sup`: found: `<sup>63</sup>`
    /html[0]/body[1]/p[0] Unexpected element: ` nanoseconds in either direction).`

mio looks like it gets "out of sync" when looking at link tags? This probably requires a closer look.

Details

 Documenting mio v0.6.10
WARNING: documentation for this crate may be rendered differently using the new Pulldown renderer.
    See https://github.com/rust-lang/rust/issues/44229 for details.
WARNING: rendering difference in `An readiness event returned by [`Poll::poll`]....`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.10/src/event_imp.rs:941:0
    /html[0]/body[1]/p[1] Text differs:
        expected: `is a`
        found:    `is a [readiness state] paired with a`
    /html[0]/body[1]/p[1] Attributes differ in `a`: expected: `{"href": "struct.Ready.html"}`, found: `{"href": "struct.Token.html"}`
    /html[0]/body[1]/p[1] Text differs:
        expected: `paired with a`
        found:    `. It is returned by`
    /html[0]/body[1]/p[1] Attributes differ in `a`: expected: `{"href": "struct.Token.html"}`, found: `{"href": "struct.Poll.html#method.poll"}`
    /html[0]/body[1]/p[1] Text differs:
        expected: `. It is returned by`
        found:    `.`
    /html[0]/body[1]/p[1] One element is missing: expected: `a`
    /html[0]/body[1]/p[1] One element is missing: expected: ``
WARNING: rendering difference in `A collection of readiness events....`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.10/src/poll.rs:1180:0
    /html[0]/body[1]/p[1] Attributes differ in `a`: expected: `{"href": "struct.Poll.html#method.poll"}`, found: `{"href": "struct.Poll.html"}`
    /html[0]/body[1]/p[1] Attributes differ in `a`: expected: `{"href": "struct.Poll.html#method.poll"}`, found: `{"href": "struct.Poll.html"}`
    /html[0]/body[1]/p[1] Attributes differ in `a`: expected: `{"href": "struct.Poll.html#method.poll"}`, found: `{"href": "struct.Poll.html"}`
    /html[0]/body[1]/p[2] Attributes differ in `a`: expected: `{"href": "struct.Poll.html#method.poll"}`, found: `{"href": "struct.Poll.html"}`
WARNING: rendering difference in `A collection of readiness events....`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-    0.6.10/src/poll.rs:1180:0
    /html[0]/body[1]/p[1] Attributes differ in `a`: expected: `{"href": "struct.Poll.html#method.poll"}`, found: `{"href": "struct.Poll.html"}`
    /html[0]/body[1]/p[1] Attributes differ in `a`: expected: `{"href": "struct.Poll.html#method.poll"}`, found: `{"href": "struct.Poll.html"}`
    /html[0]/body[1]/p[1] Attributes differ in `a`: expected: `{"href": "struct.Poll.html#method.poll"}`, found: `{"href": "struct.Poll.html"}`
    /html[0]/body[1]/p[2] Attributes differ in `a`: expected: `{"href": "struct.Poll.html#method.poll"}`, found: `{"href": "struct.Poll.html"}`
WARNING: rendering difference in `Set the registration's readiness...`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/mio-0.6.10/src/poll.rs:1614:4
    /html[0]/body[1]/p[1] Attributes differ in `a`: expected: `{"href": "struct.Poll.html"}`, found: `{"href": "struct.Poll.html#method.poll"}`
    /html[0]/body[1]/p[1] Attributes differ in `a`: expected: `{"href": "struct.Poll.html"}`, found: `{"href": "struct.Poll.html#method.poll"}`

Something similar happens in hyper as well:

Details

 Documenting hyper v0.11.2 (file:///home/misdreavus/clones/hyper)
WARNING: documentation for this crate may be rendered differently using the new Pulldown renderer.
    See https://github.com/rust-lang/rust/issues/44229 for details.
WARNING: rendering difference in `Percent encode a sequence of bytes with a character set defined in...`
   --> src/header/parsing.rs:153:0
    /html[0]/body[1]/p[0] Text differs:
        expected: `......`
        found:    `...`
    /html[0]/body[1]/p[0] Unexpected element `a`: found: `<a href="https://tools.ietf.org/html/rfc5987#secti ... 5987#section-3.2</a>`
WARNING: rendering difference in `The `Pragma` header defined by HTTP/1.0....`
   --> src/header/common/pragma.rs:34:0
    /html[0]/body[1]/p[2] Text differs:
        expected: `Spec: https://tools.ietf.org/html/rfc7234#section-5.4`
        found:    `Spec:`
    /html[0]/body[1]/p[2] Unexpected element `a`: found: `<a href="https://tools.ietf.org/html/rfc7234#secti ... 7234#section-5.4</a>`
WARNING: rendering difference in `A Media Descriptors Enum based on:...`
   --> src/header/common/link.rs:112:0
    /html[0]/body[1]/p[0] Text differs:
        expected: `......`
        found:    `...`
    /html[0]/body[1]/p[0] Unexpected element `a`: found: `<a href="https://www.w3.org/TR/html401/types.html# ... ypes.html#h-6.13</a>`
WARNING: rendering difference in `A Mime charset....`
   --> src/header/shared/charset.rs:14:0
    /html[0]/body[1]/p[2] Text differs:
        expected: `... http://www.iana.org/assignments/character-sets/character-sets.xhtml`
        found:    `...`
    /html[0]/body[1]/p[2] Unexpected element `a`: found: `<a href="http://www.iana.org/assignments/character ... acter-sets.xhtml</a>`
WARNING: rendering difference in `The `Origin` header....`
   --> src/header/common/origin.rs:35:0
    /html[0]/body[1]/p[2] Text differs:
        expected: `... https://fetch.spec.whatwg.org/#origin-header, the value of this header is composed of...`
        found:    `...`
    /html[0]/body[1]/p[2] Unexpected element `a`: found: `<a href="https://fetch.spec.whatwg.org/#origin-hea ... g/#origin-header</a>`
    /html[0]/body[1]/p[2] Unexpected element: `, the value of this header is composed of...`
WARNING: rendering difference in `The scheme, such as http or https...`
   --> src/header/common/origin.rs:76:4
    /html[0]/body[1]/p[0] Unexpected element `code`: found: `<code>use hyper::header::Origin; let origin = Orig ... me("https"));</code>`
    /html[0]/body[1] One element is missing: expected: `pre`
WARNING: rendering difference in `The host, such as Host{hostname: "hyper.rs".to_owned(), port: None}...`
   --> src/header/common/origin.rs:89:4
    /html[0]/body[1]/p[0] Unexpected element `code`: found: `<code>use hyper::header::{Origin,Host}; let origin ... Some(443))));</code>`
    /html[0]/body[1] One element is missing: expected: `pre`

I'll add more samples over time.

[QuietMisdreavus]

The results for webpki allude to the source of the link tag issue: It looks like one parser automatically linkifies things in inline code spans?

Details

 Documenting webpki v0.17.0
WARNING: documentation for this crate may be rendered differently using the new Pulldown renderer.
    See https://github.com/rust-lang/rust/issues/44229 for details.
WARNING: rendering difference in `webpki: Web PKI X.509 Certificate Validation....`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/webpki-0.17.0/src/webpki.rs:15:0
    /html[0]/body[1]/p[1]/code[0] Text differs:
        expected: `git clone https://github.com/briansmith/webpki`
        found:    `git clone`
    /html[0]/body[1]/p[1]/code[0] Unexpected element `a`: found: `<a href="https://github.com/briansmith/webpki">htt ... riansmith/webpki</a>`
WARNING: rendering difference in `RSA PSS signatures using SHA-512 for keys of 2048-8192 bits and of...`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/webpki-0.17.0/src/signed_data.rs:263:0
    /html[0]/body[1]/p[0] Text differs:
        expected: `... https://tools.ietf.org/html/rfc4055#section-1.2`
        found:    `...`
    /html[0]/body[1]/p[0] Unexpected element `a`: found: `<a href="https://tools.ietf.org/html/rfc4055#secti ... 4055#section-1.2</a>`
WARNING: rendering difference in `RSA PSS signatures using SHA-384 for keys of 2048-8192 bits and of...`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/webpki-0.17.0/src/signed_data.rs:254:0
    /html[0]/body[1]/p[0] Text differs:
        expected: `... https://tools.ietf.org/html/rfc4055#section-1.2`
        found:    `...`
    /html[0]/body[1]/p[0] Unexpected element `a`: found: `<a href="https://tools.ietf.org/html/rfc4055#secti ... 4055#section-1.2</a>`
WARNING: rendering difference in `RSA PSS signatures using SHA-256 for keys of 2048-8192 bits and of...`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/webpki-0.17.0/src/signed_data.rs:245:0
    /html[0]/body[1]/p[0] Text differs:
        expected: `... https://tools.ietf.org/html/rfc4055#section-1.2`
        found:    `...`
    /html[0]/body[1]/p[0] Unexpected element `a`: found: `<a href="https://tools.ietf.org/html/rfc4055#secti ... 4055#section-1.2</a>`

[QuietMisdreavus]

On a farther read, the webpki may be a true rendering difference: The line being rendered there doesn't use backticks or indentation: it's a manual <code> span written into the docs. This may have been a deliberate decision to have the link in the code span, but it seems that pulldown suppresses the link generation that hoedown did, even when it's a manual <code> tag.

EDIT: gonna add farther notes to this comment rather than spamming the thread.

time's rendering differences seem to all be true differences: The aforementioned <sup> change, a link anchor being spread across multiple lines (proper link in hoedown, broken tag in pulldown), and asterisks inline to a word being treated differently (hoedown left them as asterisks, pulldown created an <em> tag for them).

EDIT 2: mio is also all legit. The last three are also about the case-insensitivity of reference links. The first is a new difference: link reference tags are whitespace-stripped in Pulldown, but not Hoedown. The published 0.6.10 has no link for "readiness state" because the anchor added a space before the closing bracket. Pulldown stripped that space and completed the link, hence getting "out of sync" because there's an anchor tag where there wasn't one before.

EDIT 3: hyper is also all legit. Several URLs that aren't auto-linked in Pulldown, and a couple places where there wasn't a newline between a regular paragraph and a ``` code block.

EDIT 4: mime introduces yet another new difference:

Details

 Documenting mime v0.3.3 (file:///home/misdreavus/clones/mime)
WARNING: documentation for this crate may be rendered differently using the new Pulldown renderer.
    See https://github.com/rust-lang/rust/issues/44229 for details.
WARNING: rendering difference in `*`
   --> src/lib.rs:503:8
    /html[0]/body[1] Tags differ: expected: `ul`, found: `p`

Here, the entire docstring is a single asterisk. Hoedown leaves it alone, puts it into the output as-is. Pulldown creates a new unordered list with an single, empty item.

[QuietMisdreavus]

rocket (the crate i tried to start with before i started rummaging through its dependencies) has a fine stack of warnings:

Details

 Documenting rocket v0.3.2 (file:///home/misdreavus/clones/rocket)
WARNING: documentation for this crate may be rendered differently using the new Pulldown renderer.
    See https://github.com/rust-lang/rust/issues/44229 for details.
WARNING: rendering difference in `Application configuration and configuration parameter retrieval....`
   --> src/config/mod.rs:1:0
    /html[0]/body[1]/ul[10]/li[5] Tags differ: expected: `strong`, found: `p`
    /html[0]/body[1]/ul[10]/li[5] Types differ: expected: ``, found: `ol`
    /html[0]/body[1]/ul[10]/li[5] Tags differ: expected: `em`, found: `ul`
    /html[0]/body[1]/ul[10]/li[5] One element is missing: expected: ``
    /html[0]/body[1]/ul[10]/li[5] One element is missing: expected: `ol`
    /html[0]/body[1]/ul[10]/li[5] One element is missing: expected: `ul`
    /html[0]/body[1]/ul[10]/li[6] Tags differ: expected: `strong`, found: `p`
    /html[0]/body[1]/ul[10]/li[6] Types differ: expected: ``, found: `ul`
    /html[0]/body[1]/ul[10]/li[6] One element is missing: expected: `em`
    /html[0]/body[1]/ul[10]/li[6] One element is missing: expected: ``
    /html[0]/body[1]/ul[10]/li[6] One element is missing: expected: `em`
    /html[0]/body[1]/ul[10]/li[6] One element is missing: expected: ``
    /html[0]/body[1]/ul[10]/li[6] One element is missing: expected: `em`
    /html[0]/body[1]/ul[10]/li[6] One element is missing: expected: ``
    /html[0]/body[1]/ul[10]/li[6] One element is missing: expected: `ul`
WARNING: rendering difference in `The `Pragma` header defined by HTTP/1.0....`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.13/src/header/common/pragma.rs:34:0
    /html[0]/body[1]/p[2] Text differs:
        expected: `Spec: https://tools.ietf.org/html/rfc7234#section-5.4`
        found:    `Spec:`
    /html[0]/body[1]/p[2] Unexpected element `a`: found: `<a href="https://tools.ietf.org/html/rfc7234#secti ... 7234#section-5.4</a>`
WARNING: rendering difference in `A Mime charset....`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.13/src/header/shared/charset.rs:14:0
    /html[0]/body[1]/p[2] Text differs:
        expected: `... http://www.iana.org/assignments/character-sets/character-sets.xhtml`
        found:    `...`
    /html[0]/body[1]/p[2] Unexpected element `a`: found: `<a href="http://www.iana.org/assignments/character ... acter-sets.xhtml</a>`
WARNING: rendering difference in `The `Origin` header....`
   --> /home/misdreavus/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.10.13/src/header/common/origin.rs:34:0
    /html[0]/body[1]/p[2] Text differs:
        expected: `... https://fetch.spec.whatwg.org/#origin-header, the value of this header is composed of...`
        found:    `...`
    /html[0]/body[1]/p[2] Unexpected element `a`: found: `<a href="https://fetch.spec.whatwg.org/#origin-hea ... g/#origin-header</a>`
    /html[0]/body[1]/p[2] Unexpected element: `, the value of this header is composed of...`
WARNING: rendering difference in `Returns the Media-Type associated with the extension `ext`. Not all...`
   --> src/http/media_type.rs:168:8
    /html[0]/body[1] Tags differ: expected: `h1`, found: `p`
WARNING: rendering difference in `Media type for <b>...`
   --> src/http/media_type.rs:115:12
    /html[0]/body[1]/p[0]/i[1] Unexpected element `*
/
*
`: found: `...`
    /html[0]/body[1] One element is missing: expected: `i`
    /html[0]/body[1] One element is missing: expected: `p`
WARNING: rendering difference in `Checks if the value is an ASCII punctuation charac ... s_ascii_punctuation)`
   --> /home/misdreavus/git/rust/src/libstd/ascii.rs:448:4
    /html[0]/body[1]/p[0]/code[2] Text differs:
        expected: `[ \\ ] ^ _ \`` U+007B ... U+007E`
        found:    `[ \\ ] ^ _ \`
    /html[0]/body[1]/p[0] Types differ: expected: ``, found: `code`
    /html[0]/body[1]/p[0] Types differ: expected: `a`, found: ``
    /html[0]/body[1]/p[0] Unexpected element `a`: found: `<a href="https://doc.rust-lang.org/nightly/std/asc ... ation">Read more</a>`
WARNING: rendering difference in `An `Accept` header with the single media type for <b>...`
   --> src/http/accept.rs:157:12
    /html[0]/body[1]/p[0]/i[2] Unexpected element `*
/
*
`: found: `...`
    /html[0]/body[1] One element is missing: expected: `i`
    /html[0]/body[1] One element is missing: expected: `p`
WARNING: rendering difference in `Content-Type for <b>...`
   --> src/http/content_type.rs:54:12
    /html[0]/body[1]/p[0]/i[1] Unexpected element `*
/
*
`: found: `...`
    /html[0]/body[1] One element is missing: expected: `i`
    /html[0]/body[1] One element is missing: expected: `p`

In sum, rocket continues to deliver as the quintessential rustdoc stress test. There are a few new rendering differences in here, a couple of which are technically bugs in pulldown.

• When there is a blank line in the middle of a list, pulldown will only wrap the items after that blank line with <p> tags - this is a bug in pulldown, as the commonmark spec asks to put every item in a "loose" list in a <p> tag
  • Hoedown has similar behavior, but also puts a <p> tag around the item immediately before the blank line
• Hoedown does not treat a header marker as a heading unless it's at the beginning of the line (without leading spaces); pulldown does not care about the leading spaces (until it becomes a code block, of course)
  • Note that this fixes Markdown fails to be interpreted after #[doc] attribute is used #42760 without having to change rustdoc itself to handle it
• When there is a raw HTML tag wrapping a section of multiple lines, pulldown still treats lines with only asterisks on them as unordered lists - this is technically a bug, as the commonmark.js dingus performs the same as hoedown, which renders each asterisk plainly

[QuietMisdreavus]

I did all of this testing using the approved-but-not-yet-merged #44368. Judging from how all of these turned out, i'm willing to say that as soon as that lands, all the rendering warnings left will be true rendering differences. It doesn't mean that Pulldown is perfect, just that we're accurately talking about what it's doing.