RFC: Declarative macro metavariable expressions

[markbt]

Add new syntax to declarative macros to give their authors easy access to additional metadata about macro metavariables, such as the index, length, or count of macro repetitions.

This RFC has been drafted as part of the lang-team declarative macro repetition counts project.

Rendered

[RustyYato]

Repetition counting can be efficiently done with recursion like so: playground. Even for 100 000 elements on my 7+year old laptop compiles in ~5s. Maybe something like this should be added to std?

macro_rules! count {
    () => { 0 };
    ($odd:tt $($a:tt $b:tt)*) => {
        (count!($($a)*) << 1) | 1
    };
    ($($a:tt $even:tt)*) => {
        (count!($($a)*) << 1)
    };
}

We could make this faster with more splitting cases, for example instead of just two branches, you could do 4 branches and that get's the time down to 1.5s for 100 000 elements on my machine

macro_rules! count {
    () => { 0 };
    ($($a:tt $b:tt $c:tt $d:tt)*) => {
        count!($($a)*) << 2
    };
    ($odd:tt $($a:tt $b:tt $c:tt $d:tt)*) => {
        (count!($($a)*) << 2) | 1
    };
    ($odd_1:tt $odd_2:tt $($a:tt $b:tt $c:tt $d:tt)*) => {
        (count!($($a)*) << 2) | 2
    };
    ($odd_1:tt $odd_2:tt $odd_3:tt $($a:tt $b:tt $c:tt $d:tt)*) => {
        (count!($($a)*) << 2) | 3
    };
}

[Lokathor]

But then every single macro writer also needs to know how to do that.

and that also does not handle the other cases that this will initially supports (index, length, ignore).

[RustyYato]

But then every single macro writer also needs to know how to do that [count].

Not if it's in std (Which I think it should be)

and that also does not handle the other cases that this will initially supports (index, length, ignore).

length can be modeled based on count, and ignore can be a regular macro

macro_rules! ignore {
    ($($a:tt)*) => ();
}

This just leaves index, and I'm not sure how useful that is. In all the macros I've written I've only needed something similar a handful of times. But I will concede that it is nice. However, on it's own it doesn't carry enough weight for new syntax.

[markbt]

That macro is very clever! It doesn't match anything I found when I was trying to find how to do this, and it's a lot better in terms of performance. Since it's counting the bits in the length it has a constant recursion depth.

In an earlier draft of an RFC that targetted counts specifically (rendered), I benchmarked the ones I found in the wild. The common recursive cases caused compiler stack overflows and took 10s of seconds for large counts. The more efficient implementations (taking the length of a slice) still took 6 or more seconds for 10,000 items.

My prototype implementation was considerably faster (it's effectively instant to get the count or index, since it was just accessing internal compiler state), which is what motivated working on this RFC.

Interestingly I just repeated my benchmark with the slice-length approach and it's also much improved, so kudos to everyone working on compiler performance.

New benchmarks (time to compile a simple program that counts a repetition with 10024 items, average of 100 runs):

• Slice length (best I had previously seen): 509ms
• Metavariable expressions (this RFC): 208ms
• @RustyYato's macro: 286ms

[markbt]

More benchmarks, increasing the length to 100,251 items:

• Slice length (best I had previously seen): 3,555ms
• Metavariable expressions (this RFC): 492ms
• @RustyYato's macro: 904ms
• count_tts::count_tts!: 687ms

[markbt]

Even more benchmarks. Since most users don't count a single macro with tens of thousands of items, I figured a more reasonable benchmark would be large numbers of invocations of macros, each counting a modest number of items, simulating a large codebase with lots of uses of a macro that needs to count things. For example, the codebase I work in has hundreds of invocations of maplit::hashmap! (mostly inputs in unit tests), which is the sort of macro that could use this knowledge.

I made a new test case that counted 1,000 instances of a repetition with 101 items. The results were a bit surprising:

• Slice length (best I had previously seen): 4,134ms
• Metavariable expressions (this RFC): 1,013ms
• @RustyYato's macro: 2m 26s
• count_tts::count_tts!: 1,247ms

You can see a cut down version counting 20 x 500 in the playground. This takes around 8s to compile, although that varied quite a bit. If you add more count invocations then it typically hits the playground timeout.

[RustyYato]

Oh wow! I wasn't expecting that! In that case, it looks like the metavariable approach does carry it's weight better than I expected.

[rodrimati1992]

@markbt
How do they compare with the count_tts::count_tts proc macro timewise?

[markbt]

How do they compare with the count_tts::count_tts proc macro timewise?

count_tts is pretty good, the best so far outside of this RFC. I've updated the benchmarks above. This excludes the time taken to compile the proc macro (which was around 450ms, but would presumably only need to be done once).

However, I don't want to frame this RFC as being just about the performance of counting. These alternatives don't help so well with the index, length and ignore cases, and I think there are ergonomic reasons to consider adding new ways of getting this information, and also in providing a mechanism for other information to be provided in the future.

[petrochenkov]

IIRC, I already asked this on Zulip, but could you document what syntactic space is available for this feature (and similar features) in general?

The main issue with choosing a syntax here is that pretty much any syntax is valid on the right hand side of the macro because it represents macro output, which can be an arbitrary token stream.
So, we are lucky that the combination $ { turned out reserved.
Are any other reserved combinations that can be used for new macro features?

[Lokathor]

I believe $[ ... ] was also available

[markbt]

As @Lokathor says, this is mentioned briefly in the "Future possibilities" section at the bottom, but I will expand on it.

Currently the $ symbol can only be followed by an identifier or (, so anything else can be used in a language extension. This RFC specifies ${ .. } and $$, but $[ .. ] remains invalid, as does $ followed by any other symbol (so $@, $:, $! or similar could be used).

Additionally, metavariable expressions are intended to be extensible themselves. This RFC defines count, index, length and ignore, but future RFCs can add additional expressions of the form ${foo(fooargs...)}. Anything that fits within this pattern and can be evaluated by the macro transcriber would be a suitable candidate for another expression.

[joshtriplett]

I like this, and I like that it carves out an extensible space for future improvements to macro syntax.

[pnkfelix]

I worry slightly that ${...} and $(...) might look too similar. That might be an artifact of my font.

In particular, the its going to be tricky to give good diagnostics when a user writes $(...) when they meant to write ${...}, and vice versa. Especially if their macro body happens to refer to names like count or length

But I don't have great counter-suggestions; $[...] might be just as bad (though I do think it is easier to distinguish from $(...).) The only other counter-suggestion can think of is ${{...}}, but that might be bridge too far.

[markbt]

I see your worry. I also have some reservations about the )} cluster in ${count(ident)}.

We don't have to use a delimited block. Since currently anything other than ident or (..) is invalid after $ we could use some other sigil. Some examples:

• $:count(ident) e.g. let v = Vec::with_capacity($:count(values))
• $@count(ident) e.g. let v = Vec::with_capacity($@count(values))
• $!count(ident) e.g. let v = Vec::with_capacity($!count(values))

Using the last one as an example, this would be parsed as: $ ! <metavar-expr-function> ( <metavar-expr-args...> )

Other suggestions also welcome.

[joshtriplett]

This looks great to me.

[pnkfelix]

I was a little surprised that RFC PR #88 was not referenced in the text nor in the comments so far on this PR.

[joshtriplett]

Based on discussion in today's @rust-lang/lang meeting, I'm proposing to merge this, to gauge consensus via rfcbot (rather than, as we sometimes do, waiting until we already know what the rfcbot outcome will be before proposing it). People are still welcome to take further time to review.

@rfcbot merge

[rfcbot]

Team member @joshtriplett has proposed to merge this. The next step is review by the rest of the tagged team members:

• @cramertj
• @joshtriplett
• @nikomatsakis
• @pnkfelix
• @scottmcm

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

See this document for info about what commands tagged team members can give me.

[markbt]

I was a little surprised that RFC PR #88 was not referenced in the text nor in the comments so far on this PR.

Somehow I missed that PR when I was doing my research, so thanks for the pointer.

It makes a good point that these index numbers can be used as tuple indexes (something that isn't possible with computed values). For example, it becomes possible to write a macro like this:

macro_rules! example {
    ( $tuple:expr => $( $name:expr ),* ) => {
        $( println!("{} = {}", $name, $tuple.${index()}); )*  
    };
}

fn main() {
    let t = (1.23, "hello", 42);
    example!(t => "floating", "string", "number");
}

Which outputs:

floating = 1.23
string = hello
number = 42

Without this you'd need some kind of destructuring of the tuple, which is hard as you need names for the fields.

[scottmcm]

Are they any places where this would want to go deeper? Would it be helpful to have $$$ that expands to $$, instead of needing $$$$? (Does $$$$ work with this RFC, actually? Is exponential escaping bad, or fine because "just don't go that deep"?)

[markbt]

Going deeper is only necessary if macro definitions are multiple-times recursive (a macro that defines a macro that defines a macro), and you want to defer metavariable expansions or repetitions to the inner macros in ways that are otherwise ambiguous. The doubling up of the escape characters for each level is necessary so that at each nesting level you can represent a meta-variable whose name is stored in another meta-variable. An even number of $ followed by var (e.g. $$$$var) expands to n/2 $s followed by a literal var (e.g. $$var). An odd number of $ expands to (n-1)/2 $s followed by the expansion of $var, (e.g. if $var == foo then $$$$$var expands to $$foo).

An example of where this would be necessary in existing code is here. This code is currently using $dol as a hack for what $$ would provide, and $dol $arg would become $$$arg.

This is the same as for \-escaping in strings, and most other kinds of escaping in other languages, so it should be familiar to users.

Although sixteen $ in a row wouldn't be great, quadruply-recursive macro definitions are probably not a great idea either, and it should be possible to break the macro down into separate parts with less nesting if that does become a concern.

[scottmcm]

I agree that the workarounds for these are non-obvious, and that it's worth giving them well-known names. But I think the jump to new syntax could be better-motivated in the RFC.

For example, why not std::macro_utils::count!(...) instead of ${count(...)}? If it can be written as a macro as @RustyYato showed, that would then leave it up to an implementation to choose whether to add special compiler code to optimize it or just decide that the binary matching trick is good enough.

(I suspect it won't be too hard to convince me that syntax is worth it, but I'd still like to see it addressed in the RFC text.)

[markbt]

The workaround macros work by expanding to count!($( $value )*): i.e. the compiler must generate a sequence by expanding the repetition, re-parsing it as part of the count! macro invocation, and then computing the length. This is the redundant additional work that this RFC seeks to address.

The reason for new syntax is that these expansions occur during macro transcription, rather than as their own macro expansions. ${count(ident)} would be transcribed directly to the literal count, whereas count!(ident) in a macro body would be transcribed to count!(ident) (there is no change as the transcriber has nothing to do - it doesn't peek inside macro invocations), at which point the information about what ident means is lost and the count! macro has no knowledge about what it is counting or what context it is counting it in.

Another way to think of metavariable expressions is as "macro transcriber directives". You can then think of the macro transcriber as performing the following:

• $var => the value of var
• $( ... )* => a repetition
• ${ directive } => a special transcriber directive
• $$ => $

Perhaps describing it like this makes it a bit clearer that these are special things the transcriber to do (not necessarily limited to counts and indexes, but that is what this RFC focuses on).

We could special-case these macro invocations during transcription, but that feels like a worse solution. It would make it harder to understand what the macro transcriber is going to do with arbitrary code if you don't remember all of the special macros that don't work like other macros.

(Conversely, I think there might be existing special macros that might have been better written as metavariable expressions if they had already existed. While I haven't thought it through fully, file!(), line!() and column!() spring to mind as candidates).

[scottmcm]

This might be my ignorance of macros speaking, but does the RFC need to specify which kind of fragment they produce? Are they allowed to expand to (0+4) or 1+1+1+1 or 2*2 or 4_usize, or only to exactly 4? Is there any way I can start to depend on that expansion, like writing a macro that checks a length by only accepting a particular token?

[markbt]

It doesn't, but it would be a good change to add it. It should expand to a literal with the appropriate value and no suffix (i.e. only and exactly 4). This allows consistent use in things like stringify! and tuple indexing. Type inferencing should be able to infer the correct type for when it is used in code (and also produce an error if the value that is produced won't fit inside the target type).

[markbt]

Thanks for the review @scottmcm. I will add some text to the RFC to expand on these points (although probably not until the weekend).

[joshtriplett]

On Sun, Mar 07, 2021 at 07:48:51PM -0800, scottmcm wrote: This might be my ignorance of macros speaking, but does the RFC need to specify which kind of fragment they produce? Are they allowed to expand to `(0+4)` or `1+1+1+1` or `2*2` or `4_usize`, or only to exactly `4`? Is there any way I can start to depend on that expansion, like writing a macro that checks a length by only accepting a particular token?

I think it'd be a good idea to clarify that it expands to a single integer literal, yes.

[pnkfelix]

@rfcbot reviewed

[markbt]

We don't, but I like that this proposal does so, because that then reserves all possible syntax within the delimited block for future extensions. And it feels less line-noise-y than other alternatives, as well as more distinct from $ident.

I've kept it as ${ ... } but I've included the alternatives so that they are documented in the RFC.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[joshtriplett]

cc @rust-lang/libs

This isn't a libs RFC, but in discussions in the language team meeting, we felt that since this was introducing a new family of built-in names (similar to built-in macros), it'd be appropriate to make sure libs was aware and didn't have any concerns.

[Mark-Simulacrum]

Dropping nomination - looks like this is on track and doesn't need T-lang discussion right now.

[rfcbot]

The final comment period, with a disposition to merge, as per the review above, is now complete.

As the automated representative of the governance process, I would like to thank the author for their work and everyone else who contributed.

The RFC will be merged soon.

[nikomatsakis]

Huzzah! The @rust-lang/lang team has decided to accept this RFC. If you'd like to follow along with its development, please subscribe to the tracking issue rust-lang/rust#83527.