proc-macros should not automatically error on token groups that are "locally" syntactically correct

[AaronKutch]

My limited understanding is that TokenStreams inputs and outputs for proc-macros have to abide to some minimal level of syntax (e.x. can't have mismatched delimiters outside of string literals), but everything else should be fair game for the proc-macro to handle (please edit the title if there is a better term for what I am describing). However, I found a case where the proc-macro system will forcibly error just because Rust doesn't support it, and it doesn't involve any special delimiters.
I am adding hexadecimal, binary, etc floating point support to my awint crate and encountered this problem:

proc_macro_ex!(0x123.456_i256f128); // error: hexadecimal float literal is not supported

this is frustrating because it works if I chose some other closely related convention like 0j123.456_i256f128.

I am titling this issue more generally because I want to make sure there are not other cases like it.

[Noratrieb]

We've had this discussion before about literals that are out of range of the maximal literal, and IIRC the consensus was that these should fail to lex into tokens and therefore not work. I can't remember the exact discussion though.

[AaronKutch]

So Rust does literal verification at lex time? I could imagine it doing steps like separating "-123.0f32" into groups like "- 123 . 0 f32" but it should detect and fail on overflow at another stage. If it lexes into some canonical form early on, what it should do for cases that don't match what Rust supports, is it should continue with the tokens as raw, and then normal Rust can fail on that case later and proc-macros can do what they want.

[AaronKutch]

It has been a while since I looked at TokenTree and proc_macro::Literal. I see that you wouldn't want token groups to parse as Ident or Literal differently across versions in case we made hexadecimal floats work in plain Rust or something like that, so token groups starting with '0'-'9' would always be some kind of Literal. Is it possible for there to be some "Unknown" type in the implementation details of Literal? When Literal::from_str encounters an unknown kind of literal it just stores the chars as they are. The docs say it can currently panic and "We reserve the right to change these errors into LexErrors later.". Then Literal::to_string can simply spit out what was put in, and I don't see that we break anything proc-macro user facing. Rust's internals that are intertwined more closely with the system just have to be aware of one more "Unknown" type that they parse again more rigorously to see what canonical integer it is most likely trying to represent.

[AaronKutch]

Additionally, it looks like overflow checks are in fact happening after the lexing stage, because

let _ = ex!(-999u8);
let _ = ex!(0xfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff);

doesn't produce any errors, so it looks to me like there are definitely unnecessary restrictions in place

[dtolnay]

The discussion above about overflow does not seem germane. 0x123.456_i256f128 in macro input is not disallowed because of overflow, it is disallowed because that is a syntax that Rust has reserved for its own future use. Just like #84978 makes jkl"" not valid in macro tokens.

If 0x123.456_i256f128 were to lex as multiple macro tokens (like your example of 0j123.456_i256f128, which is 3 tokens) then it would be a breaking change for Rust to introduce hexadecimal float literal using that syntax. If it were to lex as a single token, it would be a breaking change for Rust to introduce hexadecimal float literals with syntax which diverges from that one. So until a hexadecimal float literal syntax for Rust makes it through RFC, or the lang team commits via RFC to not having such a syntax, it needs to be illegal in macro input.

[AaronKutch]

The thing about "" and # prefixes makes total sense to me because it involves delimiters, but it is unfortunate that other things have a problem because of how tokens are decided.