serde_tokenstream

This Rust crate is intended for use with macros that need bespoke configuration. It's implemented as a serde::Deserializer that operates on a proc_macro2::TokenSteam (easily converted from the standard proc_macro::TokenStream).

Say we're building an attribute proc macro that you want consumers to use like this:

#[MyMacro {
    name = "SNPP",
    owner = "Canary M Burns",
    details = {
        kind = Fission,
        year_of_opening = 1968,
    }
}]
fn some_func() {
    ...
}

It's also useful for function-like macros:

my_macro!(
    name = "SNPP",
    owner = "Hans",
    layoffs_in_alphabetical_order = [
        "Simpson, Homer"
    ]
);

The function that implements the proc macro must have two parameters (both of type proc_macro::TokenStream): attributes (the tokens with the braces that follow the name of the macro), and the item (the function, type, etc. to which the macro is applied):

#[proc_macro_attribute]
pub fn MyMacro(
    attr: proc_macro::TokenStream,
    item: proc_macro::TokenStream,
) -> proc_macro::TokenStream {
    ...
}

We'll first define the struct type that represents the configuration and derive a serde::Deserialize:

#[derive(Deserialize)]
struct Config {
    name: String,
    owner: String,
    details: ConfigDetails,
}

#[derive(Deserialize)]
struct ConfigDetails {
    kind: ConfigDetailsType,
    year_of_opening: usize,
}

#[derive(Deserialize)]
enum ConfigDetailsType {
    Coal,
    Fission,
    Hydroelectric,
}

Now we can parse attr into the Config struct with serde_tokenstream::from_tokenstream:

use proc_macro2::TokenStream;
use serde_tokenstream::from_tokenstream;

#[allow(non_snake_case)]
#[proc_macro_attribute]
pub fn MyMacro(
    attr: proc_macro::TokenStream,
    item: proc_macro::TokenStream,
) -> proc_macro::TokenStream {
    let config = match from_tokenstream::<Config>(&TokenStream::from(attr)) {
        Ok(c) => c,
        Err(err) => return err.to_compile_error().into(),
    };

    ...
}

See the serde documentation for the full range of controls that can be applied to types and their members.

Errors indicate the problematic portion of consuming code to assist the macro consumer:

#[MyMacro{
    name = "Rocinante",
    owner = "Rocicorp",
    details = {
        kind = Fusion,
        year_of_opening = 2347
    }
}]
fn deploy() {
    ...
}

error: unknown variant `Fusion`, expected one of `Coal`, `Fission`, `Hydroelectric`
 --> tests/test_err1.rs:7:16
  |
7 |         kind = Fusion,
  |                ^^^^^^

For parsing attributes nested inside an outer macro, use from_tokenstream_spanned. This function provides better span attribution for errors at the top level.

The most common use is with syn::MetaList. For example, if your macro is a derive macro:

#[derive(MyRobot)]
#[robot {
    name = "Mawhrin-Skel",
    kind = Drone,
    planet = "Eä",
}]
fn monitor() {
    ...
}

Then robot can be interpreted as a syn::MetaList instance. With that:

use serde_tokenstream::from_tokenstream;

#[derive(Deserialize)]
struct Robot {
    ...
}

#[proc_macro_derive(MyRobot, attributes(robot))]
pub fn my_robot(input: proc_macro::TokenStream) -> proc_macro::TokenStream {
    let list = /* obtain the `syn::MetaList` from the input */;

    let config = match from_tokenstream_spanned::<Robot>(
        list.delimiter.span(),
        &list.tokens
    ) {
        Ok(c) => c,
        Err(err) => return err.to_compile_error().into(),
    };
}

In some cases, it's useful to pass TokenStream values as parameters to a macro. In this case we can use the TokenStreamWrapper which is a wrapper around TokenStream that implements Deserialize or ParseWrapper which is a wrapper around syn::Parse that implements Deserialize. The latter is useful for passing in, for example, a syn::Path, or other specific entities from the syn crate.

A String field accepts either a string literal or a bare identifier, but the deserialized String doesn't record where it came from. If your macro interprets a string further (as an identifier, a file name, and so on), deserialize it as a ParseWrapper<SpannedString> instead. It accepts the same inputs and records the span of the token, so that errors about the value can point at the value itself:

#[derive(Deserialize)]
struct Config {
    module: ParseWrapper<SpannedString>,
}

let config = from_tokenstream::<Config>(&attr)?;
let module = config.module.parse::<syn::Ident>().map_err(|err| {
    syn::Error::new(
        config.module.span(),
        format!("`{}` is not a valid module name: {err}", config.module.value()),
    )
})?;

Given, for example, module = "not a module", the error is reported at "not a module". Values parsed out of a SpannedString (via parse or parse_with) also carry the span.

You may want to use the map syntax with keys that cannot be used by types such as HashMap or BTreeMap because they don't implement Hash or Ord. In those cases, you can use an OrderedMap and extract the pairs as an iterator of tuples.

Let's say we we want our "keys" to be serde_json::Values and our value to be... whatever... Strings! You can't use serde_json::Value as the key in a HashMap or BTreeMap, but we can for an OrderedMap:

let config = from_tokenstream::<OrderedMap<serde_json::Value, String>>(tokens)?;

The macro can then be invoked like this:

my_macro!(
    {
        "type" = "string",
        "format" = "uuid",
    } = "uuid::Uuid",
    {
        "type" = "string",
        "format" = "ip",
    } = "std::net::IpAddr",
);