Coalescing for line doc comments

[petrochenkov]

Right now multiple line doc comments are considered separate tokens and separate attributes:

/// A
/// B
/// C
pub struct S;

is interpreted as

#[doc = "/// A"]
#[doc = "/// B"]
#[doc = "/// C"]
pub struct S;

To mitigate the perf impact from turning normal comments into doc comments (#60930) we may want to merge multi-line line comments into a single token and a single attribute:

#[doc = "/// A\n/// B\n/// C"]
pub struct S;

[nagisa]

is interpreted as

#[doc = "/// A"]
#[doc = "/// B"]
#[doc = "/// C"]
pub struct S;

It cannot be interpreted that way because /// are not stripped by rustdoc. It probably is interpreted as

#[doc = "A"]
#[doc = "B"]
#[doc = "C"]
pub struct S;

so we don’t need the /// in the concatenated string either.

[petrochenkov]

I've just checked, /// is there both in tokens and in AST.
It looks like they are occasionally (sigh) stripped, e.g. when doc comments are passed to macros.
Rustdoc strips them as well (see with_desugared_doc).

In other words, the details need to be figured out.
Ideally, tokens should be passed around losslessly/precisely without attempts to be "helpful".
Not sure how close to that we can get backward-compatibly.

[petrochenkov]

Another incompatibility is with code doing something like this

// Passed to a macro
/// Comment 1
/// Comment 2

// Macro expects exactly two attributes
#[$meta:meta]
#[$meta:meta]

(Again, we'll need crater to figure out whether we need to provide a compatibility layer or not.)

[nagisa]

I've just checked, /// is there both in tokens and in AST.

Interesting. I tried to run cargo doc on your desugared version and the markdown processor barfed at all the /// in it.

[petrochenkov]

A bit on representing tokens produced by lexer losslessly (TLDR: I think literals do it right).

Tokens need to contain their kind and data - { kind: StrLiteral, data: `"foo"` } (quotes in the data).
The kind needs to be readily available because parser uses it all the time.
So it makes sense to remove the kind bits from the data, so there's no duplication and they cannot get inconsistent by definition - { kind: StrLiteral, data: `foo` } (no quotes in the data).
The original token still can be losslessly recovered from this representation - { kind: StrLiteral, data: `foo` } => `"foo"` .
(This is what literals in the compiler currently do.)

The same logic is applicable to doc comments, which are basically raw string literals with unusual quotes (/** + */ and /// + \n).

/** line 1
    line 2
    line 3 */

/// line

=>

{
    kind: OuterBlockDocComment,
    data: ` line 1
    line 2
    line 3 `
}
{
    kind: OuterLineDocComment,
    data: ` line`
}

(the original token still can be losslessly recovered).
(This is not what literals in the compiler currently do, but they probably should.)

---

If you look at line doc comments like that, it's kind of clear why each line is a separate token.

Keeping them separate at token level, but then concatenating later kind of defeats the purpose of optimization (it may be more work, and more interned strings).
So, the optimization requires redefining what the "closing quote" of a line doc comment is (start of the next non-line-doc-comment token?).

    /// line 1
    /// line 2
    /// line 3
    struct S;

=>

{
    kind: OuterLineDocComment,
    data: ` line 1
    /// line 2
    /// line 3
    `
}

(the original token still can be losslessly recovered).

Pretty weird.

[nnethercote]

So, the optimization requires redefining what the "closing quote" of a line doc comment is (start of the next non-line-doc-comment token?).

    /// line 1
    /// line 2
    /// line 3
    struct S;

=>

{
    kind: OuterLineDocComment,
    data: ` line 1
    /// line 2
    /// line 3
    `
}

(the original token still can be losslessly recovered).

I would have thought it would be this:

{
    kind: OuterLineDocComment,
    data: "line 1\nline 2\nline 3"
}

Here data is the processed, useful form, and we can losslessly recover the original form by adding "/// " at the start and after each newline, and appending a newline at the end. (Well, I'm not sure about the newline at the end, but it definitely feels to me like the internal /// markers should be removed.)

[nnethercote]

Another incompatibility is with code doing something like this

// Passed to a macro
/// Comment 1
/// Comment 2

// Macro expects exactly two attributes
#[$meta:meta]
#[$meta:meta]

That's very unfortunate, if it means that /// multi-line doc comments and /** multi-line doc comments are distinguishable at the language level. Which definitely complicates things.

It's worth noting that if we converted every multi-line /// doc comment in liballoc/libcore/libstd to a /** ... */ doc comment, we'd likely get a big chunk of the potential improvement.