Breaking behavior change for TokenStream containing trailing joint token

[dtolnay]

Between nightly-2020-09-05 and nightly-2020-09-06, libproc_macro stopped preserving the jointness of a trailing joint punctuation token in a TokenStream (likely #76285; @matklad @petrochenkov).

Cargo.toml:

[package]
name = "repro"
version = "0.0.0"
edition = "2018"
[lib]
proc-macro = true

src/lib.rs:

use proc_macro::{Punct, Spacing, TokenStream, TokenTree};

#[proc_macro]
pub fn m(_input: TokenStream) -> TokenStream {
    eprintln!("{:#?}", TokenStream::from(TokenTree::Punct(Punct::new(':', Spacing::Joint))));
    TokenStream::new()
}

src/main.rs

repro::m!();
fn main() {}

$  cargo +nightly-2020-09-05 check
TokenStream [
    Punct {
        ch: ':',
        spacing: Joint,
        span: #7 bytes(0..12),
    },
]

$  cargo +nightly-2020-09-06 check
TokenStream [
    Punct {
        ch: ':',
        spacing: Alone,
        span: #7 bytes(0..12),
    },
]

This breaks some real-world code such as the following, though only with old versions of some libraries as far as I have found. I don't know which behavior above is correct but I would like it to be intentionally decided so that we know what behavior to implement in proc-macro2.

// [dependencies]
// serde = { version = "=1.0.113", features = ["derive"] }
// proc-macro2 = "=1.0.4"  # (works with >=1.0.5)

use serde::Serialize;

#[derive(Serialize)]
struct S {
    #[serde(skip_serializing_if = "Option::is_none")]
    field: Option<String>,
}

fails to compile since nightly-2020-09-06:

error: failed to parse path: "Option::is_none"
 --> src/main.rs:5:35
  |
5 |     #[serde(skip_serializing_if = "Option::is_none")]
  |                                   ^^^^^^^^^^^^^^^^^

[Mark-Simulacrum]

This has also broken perf.rust-lang.org's webrender-wrench, which relied on this behavior in one of the dependencies.

@matklad -- do you think a revert makes sense perhaps? Is there a patch expected soon-ish?

[Dylan-DPC-zz]

Marking this as P-critical based on the wg-prioritization discussion