fix(lexer): use is_asscii_whitespace to find first not space pos

[bvanjoi]

close #108275

[rustbot]

r? @eholk

(rustbot has picked a reviewer for you, use r? to override)

[WaffleLapkin]

I think #108275 is about the diagnostic output and not the fact that not all ascii whitespace characters are skipped. Are we sure we want to skip more characters?

[bvanjoi]

I think #108275 is about the diagnostic output and not the fact that not all ascii whitespace characters are skipped. Are we sure we want to skip more characters?

Maybe the \0xc character should be skipped, at least from the naming of the function skip_ascii_whitespace, the change is as expected.

[eholk]

This change looks reasonable to me, but I'd love to get someone from @rust-lang/lang to weigh in. It's not clear whether we've implemented the behavior we want and described it imprecisely, or described the behavior we want but implemented it incorrectly. I'd lean towards the latter (and that's the position this PR takes), but I don't feel qualified to declare that on my own.

[bvanjoi]

Seeing that it is the code implemented in #60261, perhaps we can ask @matklad or @petrochenkov to give some advice

[matklad]

My gut feeling is that only the error message is wrong here. |b| b != b' ' && b != b'\t' && b != b'\n' && b != b'\r' is the definition of "whitespace in strings", this is documented in the reference (https://doc.rust-lang.org/reference/tokens.html#string-literals).

It might be argued that the language definition is wrong here, and that we should adjust that. It indeed feels wrong that we use different definitions of whitespace when lexing tokens and when processing innards of a string. But, given that changing this would be a backwards incompatible change, I am 0.7 sure that the outcome here would be "this is a documented peculiarity".

[matklad]

This change looks reasonable to me
or described the behavior we want but implemented it incorrectly.

FWIW, the high order bit here is that this changes the visible semantics of the language. At this point in Rust's lifetime, "changes behavior" should take precedence over "reasonableness".

[bvanjoi]

Very great explanation, I will give a PR with only the error message modified in the near future

[WaffleLapkin]

Closing, since the consensus seems to be that we only need to change the diagnostic.