Lexer: say that lifetime-like tokens can't be immediately followed by '

[mattheww]

Forms like 'ab'c are rejected, so we need some way to explain why they don't tokenise as two consecutive LIFETIME_OR_LABEL tokens.

I think the best way to do this, given the Reference's current approach, is simply to add "not immediately followed by '" to the lexer rules for the lifetime-like tokens.

That matches what the implementation (lifetime_or_char()) is doing, so it's not likely to be wrong, and this chapter already has some cases of lookahead of this sort.

It also means there can be no ambiguity between CHAR_LITERAL and these tokens (I think the intent is that we have a traditional "longest matching token wins" rule, which would give the right result here, but that isn't explicitly stated anywhere).

[ehuss]

Thanks! I'll go ahead and merge, but for the most part the reference has not done a good job of handling ambiguity and precedence in the lexer or grammar. I'm not sure this is the ultimate approach to take, since I think there are several other rules that have ambiguity.

For example, there is nothing that clarifies if x'a' is a RESERVED_TOKEN_SINGLE_QUOTE or a IDENTIFIER CHAR_LITERAL. One possibility is to have a disambiguation rule that prefers "longest match" for ambiguity. So 'a'a would be a CHAR_LITERAL because that is a longer match than two LIFETIME_TOKENs (or 'a'1 is a CHAR INTEGER, because CHAR is longer than LIFETIME). Then we wouldn't need to explicitly state these kinds of rules. But I don't know if that is the best approach.