`rustc_parser` incorrectly parses groups with `Delimiter::None`

[petrochenkov]

Such groups can currently be created by proc macros.

Reproduction, proc macro crate:

#[proc_macro]
pub fn add_mul(_: TokenStream) -> TokenStream {
    // ⟪ 2 + 2 ⟫ * 2
    vec![
        TokenTree::from(Group::new(Delimiter::None, "2 + 2".parse().unwrap())),
        TokenTree::from(Punct::new('*', Spacing::Alone)),
        TokenTree::from(Literal::u8_unsuffixed(2)),
    ].into_iter().collect()
}

User crate:

fn main() {
    let x = add_mul!();
    assert_eq!(x, 8); // FAIL: the result is 6 != 8
}

This is not a huge issue right now because proc macros have no reasons to created Delimiter::None delimiters themselves (but they can still get them from input and return them if they return their input partially or fully, see the example below).

However, this will became a huge issue when interpolated tokens like $e: expr migrate from AST pieces to token streams, because in the token stream model they are represented exactly like groups with Delimiter::None delimiters (proc macros already see them in this way).

[petrochenkov]

How to reproduce this without constructing Delimiter::None by yourself.

Proc macro crate:

#[proc_macro]
pub fn add_mul(input: TokenStream) -> TokenStream {
    let mul_2 = vec![
        TokenTree::from(Punct::new('*', Spacing::Alone)),
        TokenTree::from(Literal::u8_unsuffixed(2)),
    ];
    input.into_iter().chain(mul_2.into_iter()).collect()
}

User crate:

macro_rules! mbe {
    ($e: expr) => ( add_mul!($e) )
}

fn main() {
    let x = mbe!(2 + 2);
    assert_eq!(x, 8); // FAIL: the result is 6 != 8
}

[petrochenkov]

What it means to parse Delimiter::None correctly is a big language design question in general.

However, it's clear that in pieces of grammar with expression-like priorities they should behave more or less like parentheses to keep the "operator priority hygiene" property of our macros.

"How would an interpolated token behave here?" should probably be the primary intuition here.

[petrochenkov]

cc @dtolnay @alexcrichton
To make you aware of this issue, and perhaps proc-macro2 and syn could attempt hiding this bug somehow.

[petrochenkov]

Encountered in practice in #74036.

[nnethercote]

I have made a serious attempt to fix this issue in #114647. The commits there eliminate Nonterminal and Token::Interpolate, and stop the stripping of invisible delimiters in the parser. This basically works for declarative macros, where pasted code fragments have a known kind, such as expr, stmt, item, etc. But it's harder for proc macros, where the produced code fragments have unknown kinds. A lot of the time the kind is clear from context, but there are certain cases where it's very difficult to know how a proc-macro-produced fragment should be parsed. See this comment for details. This seems hard to work around, and could be a potential showstopper.