Tracking issue for "Macros 1.1" (RFC #1681)

[nikomatsakis]

Tracking issue for rust-lang/rfcs#1681.

cc @alexcrichton

Stabilization TODO

Litmus tests:

• - Does diesel work? Tracking issue for "Macros 1.1" (RFC #1681) #35900 (comment)
• - Does serde work? https://users.rust-lang.org/t/serde-transitioning-to-macros-1-1/7437

Features:

• Crate name, currently is proc_macro
• Crate type, currently proc-macro
• The #[proc_macro_derive(Foo)] attribute
• Loading proc-macro crates with -L and #[macro_use] to load them
• shadowing is an error
• no hygiene
• passing a token stream for the entire struct and receiving it all back
• Cargo manifest attribute, currently proc-macro = true

Known bugs:

• - Panicking derive may have the wrong span - Second panicking custom derive has bad span #36935
• - token streams with mod foo fail - Custom derives: TokenStream::from_str fails if the string contains 'mod foo;' #36691
• - docs aren't published for proc_macro - Publish docs for proc_macro crate #38749
• - custom attributes for multiple modes is hard - Tracking issue for "Macros 1.1" (RFC #1681) #35900 (comment)
• - cargo test fails for proc macro libs - 'cargo test' with proc_macro_derive fails #37480
• - order still matters - Tracking issue for "Macros 1.1" (RFC #1681) #35900 (comment) (fxed by macros: expand #[derive]s after other attribute macros and improve intra-#[derive] ordering #37067)
• - Can't document rustc-macro crates - Can't document rustc-macro crates #36820 (fixed in rustdoc: Fix documenting rustc-macro crates #36847)
• - Cargo rebuilds too often - Dep info contains invalid file paths when rustc-macro is used #36625 (fixed in rustc: Use a special filename for macros 1.1 #36776)
• - Attributes generated by the compiler make custom derive authors' lives difficult - Tracking issue for "Macros 1.1" (RFC #1681) #35900 (comment)
  • - Change order of expansions to users-first - Tracking issue for "Macros 1.1" (RFC #1681) #35900 (comment) (fixed in rustc: Tweak expansion order of custom derive #36782)

Implementation TODO

• - Create a rustc_macro crate
  • - Have librustc_macro link to libsyntax. Depend on librustc_macro in librustc_driver
  • - Tag rustc_macro as unstable with our standard header.
  • - Only tag rustc_macro with #![crate_type = "rlib"], do not produce a dylib.
  • - Implement the API of rustc_macro using libsyntax's TokenStream internally
  • - tag rustc_macro with a TokenStream lang item so the compiler knows about it.
• - Add rustc_macro_derive attribute
  • - validate it's of the exact form foo(bar), no other arguments/formats
  • - verify it's only applied to functions
  • - verify it's only applied to functions in the root module
  • - verify the signature with the TokenStream lang item added above
  • - encode all function symbols with rustc_macro_derive into metadata along with the derive mode they're used for.
• - Add a rustc-macro crate type for other crates
  • - wire it up to produce a dylib
  • - ensure the dylib gets metadata
  • - ensure rustc-macro crates cannot be linked as dylibs
  • - ensure there are no reachable items other than those tagged with #[rustc_macro_derive]
  • - Add cfg(rustc_macro) as an unstable cfg directive, set it for the rustc-macro crate type
  • - ensure that rustc-macro crates link dynamically to libsytnax
• - Fill in #[macro_use] support for rustc-macro crates
  • - extend library loader to find rustc-macro crates separately from dylib/rlib tracked today when loding crates
  • - Parse metadata for rustc-macro crates to learn about symbol/derive mode pairings
  • - dlopen the dylib
  • - generate an error if any derive mode would shadow any other.
• - Add cargo integeration
  • - recognize rustc-macro similar to plugin = true
  • - pass --crate-type=rustc-macro when depending on it
  • - plumb the same host/target logic for rustc-macro crates as is present for plugin crates (e.g. always compile rustc-macro crates for hosts)
• - Tests
  • - smoke test loading a rustc-macro, dummy #[derive] trait
  • - name conflicts are an error
  • - compiling for the wrong architecture is an error
  • - cannot emit crate type rustc-macro and anything else (e.g. rustc-macro+dylib) is an error
  • - span information not being horrendous
  • - removing attributes from a struct, as well as fields
  • - adding impls next to a struct
  • - cross compilation looks for the host rustc-macro crate type
  • - don't load vanilla dylibs as rustc-macro crate types
  • - can't have public exports beyond macro derive functions in a rustc-macro crate
  • - derive macros must have required signature
  • - load two macro crates in one compilation

[alexcrichton]

I've updated the issue description with a checklist of what needs to be done. It's likely not exhaustive but it should get us 90% of the way there hopefully

[alexcrichton]

Ok, I'm going to look into this today and see how far I get.

[alexcrichton]

cc #35957, an implementation

[nrc]

From #35957: we should bikeshed the name of the librustc_macro crate some more. In particular, this is intended to be a long-lived crate which will have essentials for all macro authors in it, so limiting to rustc_macro (which in my mind, at least) is just about the 1.1 idea seems bad. I'd previously wanted libmacro for this, but since macro is a reserved word (and we might want it as a keyword in the future) that is impossible. @cgswords and I have been using libproc_macro, and I think that is not a bad name, although I am not 100% happy with it.

[eternaleye]

@nrc: Okay, my immediate thoughts on naming:

• extern crate macros; - short and sweet, but might be read as containing macros, rather than support code for writing them
• extern crate macro_runtime; - exactly what it says on the tin
• extern crate metarust; - write Rust, about Rust, for operating on Rust
• extern crate bikeshed; - with procedural macros, you can have whatever color of Rust you want!
• extern crate macrame; - sounds like "macro make[r]"; possibly better left for a future "nice" API over the raw library.

[aturon]

@nrc It seems like an important aspect of this question is the naming of our various macro styles over all -- in particular, if we go with libproc_macro, we're making a hard commitment to the "procedural macro" terminology. I don't have a strong opinion here, but I'm not sure if we've openly explored the space of terminology.

To be clear, are you thinking that today's macro_rules would simply be "macros", i.e. the default thing we mean by macros, while you have to qualify "procedural macros"? That seems like a pretty reasonable plan to me. And in that world, I'd argue that libproc_macro is better than libmacros.