Matching `tt` is almost useless

[paulstansifer]

Because of #3232, the macro parser can't match syntax of the form $($x:some_nt)* under any circumstances. However, this is the natural way to deal with token trees, and adding a delimiter of some kind is prohibitively unergonomic.

As a short-term fix, we could special-case tt so that the macro parser doesn't farm it out to the main Rust parser. We could also introduce a ttseq or something to mean "zero or more tts". It's conceivable that we could also try being smarter with lookahead, but I don't think that's going to be fruitful.

[vadimcn]

This came up as I was trying to parse Rust syntax using macros:

macro_rules! stmt_list(
    ( while $cond:expr { $($body:tt)+ } ) => ( while $cond { $($body:tt)+ } );
    ( $head:stmt ; ) => ( { $head; } );
    ( $head:stmt ; $($rest:tt)+ ) => ( { $head ; stmt_list!( $($rest)+ ) } );
)

fn main()
{
    trace_macros!(true);

    stmt_list!(
        while x < 100 {
            let y = x + 1;
            x = y;
        }
    )
}

>rustc --pretty expanded parse.rs
stmt_list! { while y < 0 { let y = x + 1 ; x = y ; } }
parse.rs:37:8: 37:9 error: Local ambiguity: multiple parsing options: built-in NTs tt ('body') or 1 other options.
parse.rs:37         }
                    ^

[vadimcn]

I suppose the other way to fix this would be to allow passing interpolated stms as macro parameters. I could then write my statement sequence pattern as
( $head:stmt ; $($rest:stmt);+ ) => ( { $head ; stmt_list!( $($rest);+ ) } );

[paulstansifer]

@vadimcn Passing interpolated statements is allowed; the problem is that they cannot be broken apart and re-parsed after they have been parsed once.

[steveklabnik]

Triage: updated code:

#![feature(trace_macros)]

macro_rules! stmt_list(
    ( while $cond:expr { $($body:tt)+ } ) => ( while $cond { $($body:tt)+ } );
    ( $head:stmt ; ) => ( { $head; } );
    ( $head:stmt ; $($rest:tt)+ ) => ( { $head ; stmt_list!( $($rest)+ ) } );
);

fn main()
{
    trace_macros!(true);

    stmt_list!(
        while x < 100 {
            let y = x + 1;
            x = y;
        }
    )
}

updated error and output:

stmt_list! { while x < 100 { let y = x + 1 ; x = y ; } }
hello.rs:4:64: 4:72 error: unexpected token: `an interpolated tt`
hello.rs:4     ( while $cond:expr { $($body:tt)+ } ) => ( while $cond { $($body:tt)+ } );
                                                                          ^~~~~~~~

[steveklabnik]

/cc @kmcallister

[kmcallister]

$($x:tt)* definitely does work; it's a common pattern for forwarding arguments between macros. e.g.

macro_rules! foo {
    ($($x:tt)*) => (println!($($x)*));
}

fn main() {
    foo!("Hello, {}!", "world");
}

The example originally given by @vadimcn should not have $body:tt on the right hand side of a =>, but that doesn't change the outcome. I think what @steveklabnik sees now is a completely different bug, however. The parser shouldn't ever try to process an interpolated tt as a token. It should crack it open and process the tokens inside. I'm going to look into this more.

It seems like the issue @paulstansifer originally raised here was fixed at some point, although I'm not sure.

[vadimcn]

Yeah, currently, interpolated tokens on the RHS can only be used as input to other macros.

The parsing problem on the LHS (which is what this issue is about) had been mitigated, - now it is at least possible to match $($x:tt)* when it is followed by a "close" delimiter, i.e.}, ) or ]. I think it would be very hard to fix it in the general case because of inherent parsing ambiguities.

[kmcallister]

Okay, cool. In that case I'm closing this as a dupe of #3232.