Decouple proc_macro from the rest of the compiler.#49219

eddyb · 2018-03-20T21:56:23Z

This PR removes all dependencies of proc_macro on compiler crates and allows multiple copies of proc_macro, built even by different compilers (but from the same source), to interoperate.

Practically, it allows:

running proc macro tests at stage1 (I moved most from -fulldeps to the regular suites)
using proc macros in the compiler itself (may require some rustbuild trickery)

On the server (i.e. compiler front-end) side:

server::* traits are implemented to provide the concrete types and methods
- the concrete types are completely separated from the proc_macro public API
- the only use of the type implementing Server is to be passed to Client::run

On the client (i.e. proc macro) side (potentially using a different proc_macro instance!):

client::Client wraps around client-side (expansion) function pointers
- it encapsulates the proc_macro instance used by the client
- its run method can be called by a server, to execute the client-side function
  - the client instance is bridged to the provided server, while it runs
  - ~~currently a thread is spawned, could use process isolation in the future~~
    (not the case anymore, see Run proc macro invocations in separate threads. #56058)
proc macro crates get a generated static holding a &[ProcMacro]
- this describes all derives/attr/bang proc macros, replacing the "registrar" function
- each variant of ProcMacro contains an appropriately typed Client<fn(...) -> ...>

proc_macro public APIs call into the server via an internal "bridge":

only a currently running proc macro Client can interact with those APIs
- server code might not be able to (if it uses a different proc_macro instance)
  - however, it can always create and run its own Client, but that may be inefficient
the bridge uses serialization, C ABI and integer handles to avoid Rust ABI instability
each invocation of a proc macro results in disjoint integers in its proc_macro handles
- this prevents using values of those types across invocations (if they even can be kept)

r? @alexcrichton cc @jseyfried @nikomatsakis @Zoxc @thepowersgang

eddyb · 2018-03-20T21:56:36Z

@bors try

bors · 2018-03-20T21:56:41Z

🔒 Merge conflict

eddyb · 2018-03-20T22:01:25Z

@bors try

bors · 2018-03-20T22:01:35Z

⌛ Trying commit 6e63e8d with merge c4a77e1...

@alexcrichton

Decouple proc_macro from the rest of the compiler. This PR removes all dependencies of `proc_macro` on compiler crates and allows multiple copies of `proc_macro`, built even by different compilers (but from the same source), to interoperate. On the compiler (aka "frontend") side: * `Registry` is implemented (as until now) * instead of function pointers, the `Expand{1,2}` wrapper types are received * `FrontendInterface` is implemented to provide the concrete type and methods * the concrete types are completely separated from the `proc_macro` public API * the only use of the implementer is to be passed to `Expand{1,2}::run` On the proc macro side (potentially using a different `proc_macro` instance): * `&mut Registry` is passed to the registrar (as until now) * `Expand{1,2}` wrappers are created to be passed to the `Registry` * they encapsulate the `proc_macro` instance used by the macro crate * their `run` method will set up the "current frontend" of that instance `proc_macro` public APIs call into the "current frontend" via the internal `bridge`: * only a currently running proc macro can interact with those APIs * the frontend itself might not be able to (if it uses a different `proc_macro` instance) * the `bridge` uses C ABI to avoid Rust ABI instability and "generation tagging" for safety * each invocation of a proc macro results in a different "generation" * specifically, opaque `proc_macro` types wrapping concrete frontend types are tagged * this prevents using values of those types across invocations (even if they can be kept) * an ABI mismatch between two versions of `proc_macro` is only possible during stage1 * specifically, rustc compiled by stage0 but proc macros compiled by stage1 * can only affect running tests at stage1 or the compiler using proc macros * defficiencies in the `bridge` can be addressed without waiting for a release cycle r? @alexcrichton cc @jseyfried @nikomatsakis @Zoxc @thepowersgang

eddyb · 2018-03-20T22:12:12Z

cc @rust-lang/infra We shouldn't merge this without a crater run, although you might want to wait on a review as well (then again, the results could inform the review, so whichever comes first).

bors · 2018-03-21T00:23:15Z

☀️ Test successful - status-travis
State: approved= try=True

Zoxc · 2018-03-21T01:49:05Z

src/libproc_macro/lib.rs

+
+    #[unstable(feature = "proc_macro_internals", issue = "27812")]
+    #[doc(hidden)]
+    pub contents: Term,


Do we want to commit to using interning for literals? I'm not sure interning it terrible useful for that.

Wait, I can make these fields private again. Definitely not considering stabilizing any of this.

Zoxc · 2018-03-21T02:39:54Z

src/libtest/Cargo.toml

 term = { path = "../libterm" }
+
+# not actually used but needed to always have proc_macro in the sysroot
+proc_macro = { path = "../libproc_macro" }


This should probably be a rustbuild change.

bjorn3 · 2018-03-21T17:26:46Z

Could you use something like bincode for data bridging, instead of 800 lines of own code?

eddyb · 2018-03-22T01:09:05Z

@bjorn3 I'm not serializing the data (but rather using handles), and bincode (like any other serialization) is annoying to use without derived impls (which you can't have because this is proc_macro itself - or at least I don't want to try to make that work, for now).

I am going to look into simplifying the bridge code, removing most unsafe, and making it more amenable to message-passing, however.

bjorn3 · 2018-03-22T07:11:16Z

I get it.

Mark-Simulacrum · 2018-03-22T22:38:34Z

Not sure if you wanted perf, or whether this is even helpful, but http://perf.rust-lang.org/compare.html?start=75af15ee6ca0c12b699a17984b033363cd25e9c3&end=c4a77e121949c30306dbbf48259a49c3aeb55028&stat=instructions%3Au

eddyb · 2018-03-23T13:39:34Z

@Mark-Simulacrum It doesn't look bad, that's for sure, thanks! If we switch to spawning a thread (or even a process) per invocation it'd warrant another perf run.

A few tests related to proc-macros fail when performing stage-1 testing. This PR adds // ignore-stage1 to them, along with a FIXME. Original issue: rust-lang#49352 This should (hopefully) be fixed when rust-lang#49219 is merged.

aidanhs · 2018-03-26T12:13:50Z

Crater run started

aidanhs · 2018-03-31T23:31:22Z

Hi @eddyb (crater requester), @Zoxc (PR reviewer)! Crater results are at: http://cargobomb-reports.s3.amazonaws.com/pr-49219/index.html. 'Blacklisted' crates (spurious failures etc) can be found here. If you see any spurious failures not on the list, please make a PR against that file.

(interested observers: Crater is a tool for testing the impact of changes on the crates.io ecosystem. You can find out more at the repo if you're curious)

petrochenkov · 2018-11-30T11:00:41Z

I can't believe you've done this!

rust-highfive assigned alexcrichton Mar 20, 2018

rust-highfive added the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 20, 2018

eddyb force-pushed the proc-macro-decouple branch from 67f317e to 6e63e8d Compare March 20, 2018 21:59

eddyb added the S-waiting-on-crater Status: Waiting on a crater run to be completed. label Mar 20, 2018

Zoxc reviewed Mar 21, 2018

View reviewed changes

This comment has been minimized.

Sign in to view

eddyb changed the title ~~Decouple proc_macro from the rest of the compiler.~~ [WIP] Decouple proc_macro from the rest of the compiler. Mar 22, 2018

eddyb force-pushed the proc-macro-decouple branch from 6e63e8d to cd9ac21 Compare March 22, 2018 20:07

This comment has been minimized.

Sign in to view

eddyb force-pushed the proc-macro-decouple branch from cd9ac21 to 76590d8 Compare March 23, 2018 19:13

hanna-kruppe mentioned this pull request Mar 25, 2018

Stage 1 test failure on master branch #49352

Closed

Phlosioneer mentioned this pull request Mar 25, 2018

Ignore proc macro stage 1 tests #49366

Closed

shepmaster removed the S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. label Mar 30, 2018

eddyb mentioned this pull request Mar 30, 2018

Reimplement public interface for stability dtolnay/proc-macro2#71

Merged

eddyb deleted the proc-macro-decouple branch November 30, 2018 11:13

konstin mentioned this pull request Dec 1, 2018

Find a way to update serde_derive mozilla/cbindgen#203

Closed

petrochenkov mentioned this pull request Dec 8, 2018

Panic iterating over proc_macro::TokenStream #55640

Closed

hcpl mentioned this pull request Dec 9, 2018

how to create an ident from a string dtolnay/syn#546

Closed

petrochenkov mentioned this pull request Dec 18, 2018

Remove TokenStream::JointTree. #56964

Merged

dtolnay mentioned this pull request Jan 2, 2019

Quote is hard to use for non-procmacro code generation dtolnay/quote#90

Closed

eddyb mentioned this pull request Mar 14, 2019

RFC: Stabilize the alloc crate rust-lang/rfcs#2480

Merged

nikomatsakis mentioned this pull request Mar 25, 2019

Query Parallelization Tracking Issue #48685

Open

31 tasks

Xanewok mentioned this pull request Apr 17, 2019

Use crates.io version of rustc-serialize #40527

Closed

ehuss mentioned this pull request Jul 3, 2020

Remove some ignore-stage1 annotations. #73981

Merged

mystor mentioned this pull request Jul 2, 2021

Optimize and improve the proc_macro RPC interface for cross-thread execution #86816

Closed

vlad20012 mentioned this pull request Jan 28, 2022

Add a way to disable dll copying for users of proc_macro_srv library rust-lang/rust-analyzer#11365

Merged

dtolnay mentioned this pull request May 25, 2023

Extern crate proc_macro available on all wasm targets dtolnay/proc-macro2#388

Merged

Uh oh!

Conversation

eddyb commented Mar 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eddyb commented Mar 20, 2018

Uh oh!

bors commented Mar 20, 2018

Uh oh!

eddyb commented Mar 20, 2018

Uh oh!

bors commented Mar 20, 2018

Uh oh!

eddyb commented Mar 20, 2018

Uh oh!

bors commented Mar 21, 2018

Uh oh!

Zoxc Mar 21, 2018

Choose a reason for hiding this comment

Uh oh!

eddyb Mar 21, 2018

Choose a reason for hiding this comment

Uh oh!

Zoxc Mar 21, 2018

Choose a reason for hiding this comment

Uh oh!

bjorn3 commented Mar 21, 2018

Uh oh!

eddyb commented Mar 22, 2018

Uh oh!

bjorn3 commented Mar 22, 2018

Uh oh!

This comment has been minimized.

This comment has been minimized.

Mark-Simulacrum commented Mar 22, 2018

Uh oh!

eddyb commented Mar 23, 2018

Uh oh!

aidanhs commented Mar 26, 2018

Uh oh!

aidanhs commented Mar 31, 2018

Uh oh!

petrochenkov commented Nov 30, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

eddyb commented Mar 20, 2018 •

edited

Loading