Inconsistency between Display and IntoIterator for a TokenStream containing a module

[dtolnay]

The following crate reproduces the issue. Given an attribute macro on a mod item like #[...] mod module;, iterating over the tokens of the TokenStream produces mod module; but invoking to_string() on the TokenStream produces mod module { /* the contents */ }. I don't know which one is correct but they should be consistent.

@jseyfried

---

Cargo.toml

[package]
name = "repro"
version = "0.0.0"
publish = false

[lib]
proc-macro = true

src/main.rs

#![feature(proc_macro)]

extern crate repro;

#[repro::print]
mod module;

fn main() {}

src/module.rs

pub struct S;

src/lib.rs

#![feature(proc_macro)]

extern crate proc_macro;
use proc_macro::TokenStream;

#[proc_macro_attribute]
pub fn print(_args: TokenStream, input: TokenStream) -> TokenStream {
    println!("{}", input);
    for tt in input {
        println!("{:?}", tt.kind);
    }
    TokenStream::empty()
}

Output

mod module {
    pub struct S;
}
Term(Term(mod))
Term(Term(module))
Op(';', Alone)

[alexcrichton]

I believe this is the same as #48644 as @dtolnay points out, and I've got some more explanation below that.

For this, however, there's the question of which representation is correct? Is mod module; correct or mod module { .. } correct to pass to the procedural macro attribute? Or should we forbid attributes on modules altogether?

[CodeSandwich]

I think, that whatever it is, the effect of annotation of mod module; should be same as mod module { ... }, that's what macro user would expect since in all the other uses both versions act perfectly same.

I also think, that annotation of modules WITH their content is an important feature, but my opinion could be biased since I'm the author of a crate, which uses this.

[CodeSandwich]

Since rustc 1.30.0-nightly (63d51e89a 2018-09-28) the behavior is consistent, it's always mod module;. Is there any way to access the file module's content in proc macro?