Repository navigation
Macros 2.0: #[cfg_attr] makes .to_string() and TokenStream disagree #48644
Description
I ran across a very curious case today... Given this procedural macro:
#![crate_type = "proc-macro"]
#![feature(proc_macro)]
extern crate proc_macro;
use proc_macro::TokenStream;
#[proc_macro_derive(Foo, attributes(foo))]
pub fn foo(input: TokenStream) -> TokenStream {
println!("{}", input.to_string());
print_tts(input, 0);
TokenStream::empty()
}
fn print_tts(input: TokenStream, tab: usize) {
let mut t = String::new();
for _ in 0..tab {
t.push_str(" ");
}
for token in input {
match token.kind {
proc_macro::TokenNode::Group(d, others) => {
println!("{}{:?}", t, d);
print_tts(others.into(), tab + 1);
}
s => println!("{}{:?}", t, s),
}
}
}and this expansion:
#![crate_type = "rlib"]
#![feature(proc_macro)]
#[macro_use]
extern crate foo;
#[derive(Foo)]
pub struct MyStructc {
#[cfg_attr(my_cfg, foo)]
_a: i32,
}when compiled I get:
$ rustc +nightly foo.rs && rustc +nightly bar.rs -L .
pub struct MyStructc {
_a: i32,
}
Term(Term(pub))
Term(Term(struct))
Term(Term(MyStructc))
Brace
Op('#', Alone)
Bracket
Term(Term(cfg_attr))
Parenthesis
Term(Term(my_cfg))
Op(',', Alone)
Term(Term(foo))
Term(Term(_a))
Op(':', Alone)
Term(Term(i32))
Op(',', Alone)
which is quite curious!
The to_string() representation has the field present (even though the --cfg isn't supplied) and the token stream has the actual #[cfg_attr] there.
I was actually expecting neither outcome to happen (how naive of me!) in terms of -- er, just wrong thinking here#[cfg] processing typically happening before custom derives, but maybe that's not possible?
cc @dtolnay
cc @jseyfried
cc @nrc
Activity
dtolnay commentedon Mar 1, 2018
Seems similar to how
modalso causes an inconsistency. #47627alexcrichton commentedon Mar 2, 2018
Ok I think I see what's going on here.
This has to do with this comment introduced in #43230. Basically I think this boils down to #43081, we fundamentally cannot losslessly tokenize an AST item into a list of tokens to hand to a procedural macro.
What's happening here is that all the tokens of the
struct MyStructcare saved off in the parser. This is the original source token stream which produced the struct. Later on when we tokenize the struct to hand off to a procedural macro we will use this cached token stream, if available, and tokenize it out and return it. When stringifying, however, we use the actual AST node itself (the source of truth).The bug is that cfg processing does indeed happen before the procedural macro runs. The cfg processing is modifying the AST item's internal fields but it's not invalidating the cached token stream or modifying it. Hence when the procedural macro runs the cached token stream is invalid wrt the internal state of the AST node, causing two different results to come out depending on how you look at it.
Possible solutions here are:
None of them seem like great solutions :(
proc_macro: Avoid cached TokenStream more often
Rollup merge of rust-lang#49852 - alexcrichton:fix-more-proc-macros, …