Repository navigation
rustc_parser incorrectly parses groups with Delimiter::None #67062
Description
Such groups can currently be created by proc macros.
Reproduction, proc macro crate:
#[proc_macro]
pub fn add_mul(_: TokenStream) -> TokenStream {
// ⟪ 2 + 2 ⟫ * 2
vec![
TokenTree::from(Group::new(Delimiter::None, "2 + 2".parse().unwrap())),
TokenTree::from(Punct::new('*', Spacing::Alone)),
TokenTree::from(Literal::u8_unsuffixed(2)),
].into_iter().collect()
}User crate:
fn main() {
let x = add_mul!();
assert_eq!(x, 8); // FAIL: the result is 6 != 8
}This is not a huge issue right now because proc macros have no reasons to created Delimiter::None delimiters themselves (but they can still get them from input and return them if they return their input partially or fully, see the example below).
However, this will became a huge issue when interpolated tokens like $e: expr migrate from AST pieces to token streams, because in the token stream model they are represented exactly like groups with Delimiter::None delimiters (proc macros already see them in this way).
Activity
petrochenkov commentedon Dec 5, 2019
How to reproduce this without constructing
Delimiter::Noneby yourself.Proc macro crate:
User crate:
petrochenkov commentedon Dec 5, 2019
What it means to parse
Delimiter::Nonecorrectly is a big language design question in general.However, it's clear that in pieces of grammar with expression-like priorities they should behave more or less like parentheses to keep the "operator priority hygiene" property of our macros.
"How would an interpolated token behave here?" should probably be the primary intuition here.
petrochenkov commentedon Dec 6, 2019
cc @dtolnay @alexcrichton
To make you aware of this issue, and perhaps proc-macro2 and syn could attempt hiding this bug somehow.
petrochenkov commentedon Jul 4, 2020
Encountered in practice in #74036.
Remove hacks in `make_token_stream`.
Rollup merge of rust-lang#96543 - nnethercote:rm-make_token_stream-ha…
Rollup merge of rust-lang#96543 - nnethercote:rm-make_token_stream-ha…
nnethercote commentedon Aug 21, 2023
I have made a serious attempt to fix this issue in #114647. The commits there eliminate
NonterminalandToken::Interpolate, and stop the stripping of invisible delimiters in the parser. This basically works for declarative macros, where pasted code fragments have a known kind, such asexpr,stmt,item, etc. But it's harder for proc macros, where the produced code fragments have unknown kinds. A lot of the time the kind is clear from context, but there are certain cases where it's very difficult to know how a proc-macro-produced fragment should be parsed. See this comment for details. This seems hard to work around, and could be a potential showstopper.Auto merge of rust-lang#119673 - petrochenkov:dialoc5, r=compiler-err…
8 remaining items
Added regression test for issue rust-lang#67062
Added regression test for issue rust-lang#67062
Added regression test for issue rust-lang#67062
rust: cpufreq: suppress clippy::double_parens in Policy doctest
rust: cpufreq: suppress clippy::double_parens in Policy doctest
Port Rust code and tests to Kotlin with documentation updates (#9)
Flatten/2026 05 09 (#10)
Flatten/2026 05 09 (#11)
Jules-Bertholet commentedon Jul 19, 2026
Like any miscompilation, I believe this should be considered @rustbot label I-miscompile I-unsound
rustbot commentedon Jul 19, 2026
Assigning P-high (discussion on Zulip).