Tracking Issue for proc_macro::ToTokens

Feature gate: #![feature(proc_macro_totokens)]

This is a tracking issue for adding a ToTokens trait in proc_macro, which can then be used in proc_macro::quote!. See the ACP for motivation.

Public API

This will be similar to quote::ToTokens. That can be used as a reference for implementation details since it already provides all of these.

// proc_macro

pub trait ToTokens {
    fn to_tokens(&self, tokens: &mut TokenStream);
    fn to_token_stream(&self) -> TokenStream { ... }
    fn into_token_stream(self) -> TokenStream
       where Self: Sized { ... }
}

// Aggregate token types
impl ToTokens for TokenTree { /* ... */ }
impl ToTokens for TokenStream { /* ... */ }

// Single token types
impl ToTokens for Literal { /* ... */ }
impl ToTokens for Ident { /* ... */ }
impl ToTokens for Punct { /* ... */ }
impl ToTokens for Group { /* ... */ }

// Indirect types
impl<T: ToTokens + ?Sized> ToTokens for &T { /* ... */ }
impl<T: ToTokens + ?Sized> ToTokens for &mut T { /* ... */ }
impl<T: ToTokens + ?Sized> ToTokens for Box<T> { /* ... */ }
impl<T: ToTokens + ?Sized> ToTokens for Rc<T> { /* ... */ }
impl<T: ToTokens> ToTokens for Option<T> { /* ... */ }
impl<T: ToTokens + ToOwned + ?Sized> ToTokens for Cow<T> { /* ... */ }

// Types that can create `Literal`s
impl ToTokens for {u,i}{8,16,32,64,128} { /* ... */ }
impl ToTokens for f{16,32,64,128} { /* ... */ }
impl ToTokens for bool { /* ... */ }
impl ToTokens for char { /* ... */ }
impl ToTokens for str { /* ... */ }
impl ToTokens for String { /* ... */ }
impl ToTokens for CStr { /* ... */ }
impl ToTokens for CString { /* ... */ }

/* migrate the following APIs, if possible without breakage */

// currently `Extend<TokenStream>` and `Extend<TokenTree>`
impl Extend<T: ToTokens> for TokenStream { /* ... */ }

// currently `FromIterator<TokenStream>` and `FromIterator<TokenTree>`
impl FromIterator<T: ToTokens> for TokenStream { /* ... */ }

Steps / History

ACP: ACP: add proc_macro::ToTokens to support proc_macro::quote libs-team#431
Implementation in proc_macro: Add a new trait proc_macro::ToTokens #131441
Use in proc_macro::quote!: proc_macro: Use ToTokens trait in quote macro #134693
Update proc_macro::quote! to use these traits: #...
Final comment period (FCP)¹
Stabilization PR

Unresolved Questions

What should this be named? ToTokens doesn't seem quite accurate, but I don't know what would be better (ToTokenStream? ExtendTokenStream? Those seem a bit clunky).
Considering impl<T: ToTokens> ToTokens for T is provided, should to_tokens take self by value rather than by reference so cloning isn't always necessary? (fn to_tokens(self, tokens: &mut TokenStream))

https://std-dev-guide.rust-lang.org/feature-lifecycle/stabilization.html ↩

I haven't had a chance to work on this yet and probably won't for a little while, so somebody else is welcome to pick this up. Should be pretty easy so I'll label it as such.

I think a lot of those impls need ?Sized (I'm assuming you meant for the implementer to infer that, but just in case it isn't clear):

// Indirect types
impl<T: ToTokens + ?Sized> ToTokens for &T { /* ... */ }
impl<T: ToTokens + ?Sized> ToTokens for &mut { /* ... */ }
impl<T: ToTokens + ?Sized> ToTokens for Box<T> { /* ... */ }
impl<T: ToTokens + ?Sized> ToTokens for Rc<T> { /* ... */ }
impl<T: ToTokens> ToTokens for Option<T> { /* ... */ }
impl<T: ToTokens + ?Sized> ToTokens for Cow<T> { /* ... */ }

Thanks, I updated the top post. I hadn't thought about that :)

@rustbot

@rustbot claim

BTW, the link in the section "Public API" should be quote::ToTokens instead of proc_macro::ToTokens.

I have opened a PR #131441 for the initial implementation.

Considering impl<T: ToTokens> ToTokens for T is provided, should to_tokens take self by value rather than by reference so cloning isn't always necessary? (fn to_tokens(self, tokens: &mut TokenStream))

I have the same concerns about the unnecessary clone, and commented in #131441 (comment). In the current quote! macro, we have ownership of T. I think taking it by value would be better.

If this change will be applied, a more reasonable definition of the trait might be

trait IntoTokenStream {
    fn into_token_stream(self) -> TokenStream;

    fn extend_token_stream(self, tokens: &mut TokenStream) {
        tokens.extend_one(self.into_token_stream());
    }
}

Just a quick thought.

I think taking by value probably makes sense too, and then just cloning for the &T impls. But I think it would be best to (1) finish #131441 as-is, (2) update pm::quote to use it and add tests to make sure it does what we want, (3) after we have that, try a PR that changes &self to self and see if anything unexpected happens.

unless the macro auto-clones as necessary, ToTokens taking self by value would make it very difficult to use for repeated expansions like:

let ty: TokenStream = ...;
let fields: Vec<Ident> = ...;
quote::quote! {
    struct S {
        #(#fields: #ty,)* // calls ty.to_tokens() once for every field
    }
}

Rollup merge of rust-lang#131441 - SpriteOvO:proc-macro-to-tokens-tra…

Unrolled build for rust-lang#131441

@SpriteOvO

@SpriteOvO are you still interested in working on this? The expansion of proc_macro::quote! at https://github.com/rust-lang/rust/blob/13b77c687c0f5ad06a579af6d0787fe562747501/library/proc_macro/src/quote.rs still needs to be updated (no worries if not, I'll just unassign you).

@tgross35

@tgross35 Oh, I missed that step.. I'm still willing to work on it, just a bit busy recently, probably this week or next.

Rollup merge of rust-lang#134693 - SpriteOvO:proc-macro-use-to-tokens…

Unrolled build for rust-lang#134693

Rollup merge of rust-lang#134693 - SpriteOvO:proc-macro-use-to-tokens…

Navigation Menu

Repository navigation

Tracking Issue for `proc_macro::ToTokens` #130977

Public API

Steps / History

Unresolved Questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Tracking Issue for proc_macro::ToTokens #130977

Description

Public API

Steps / History

Unresolved Questions

Footnotes

Activity

tgross35 commented on Sep 28, 2024

programmerjake commented on Sep 29, 2024

tgross35 commented on Sep 29, 2024

SpriteOvO commented on Sep 29, 2024

SpriteOvO commented on Oct 9, 2024

tgross35 commented on Oct 12, 2024

programmerjake commented on Oct 13, 2024

tgross35 commented on Dec 17, 2024

SpriteOvO commented on Dec 17, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Tracking Issue for `proc_macro::ToTokens` #130977