`html.parser(convert_charrefs=False)` silently drops ampersand (&) on invalid named entities, causing data loss

[T90REAL]

Bug report

Bug description:

When HTMLParser is initialized with convert_charrefs=False, it behaves incorrectly when processing an invalid named entity reference (e.g., &A, which is not a valid HTML entity). The parser silently drops the & character and only passes the subsequent A to handle_data. I think this indicates a silent data loss problem.

from html.parser import HTMLParser

class MyParser(HTMLParser):
    def handle_data(self, data):
        print(f"handle_data received: {data!r}")

parser_false = MyParser(convert_charrefs=False)
parser_false.feed('&A')
parser_false.close()

handle_data received: 'A'

CPython versions tested on:

3.12

Operating systems tested on:

Linux

Linked PRs

• gh-140875: Fix handling of unclosed charrefs before EOF in HTMLParser #140904
• [3.14] gh-140875: Fix handling of unclosed charrefs before EOF in HTMLParser (GH-140904) #141745
• [3.13] gh-140875: Fix handling of unclosed charrefs before EOF in HTMLParser (GH-140904) #141746

[T90REAL]

I've discovered that when processing an invalid entity followed by other characters (e.g., '&A/'), the parser incorrectly calls handle_entityref('A') instead of passing '&A/' to handle_data. This probably indicates some flaw in the parser's state machine when convert_charrefs=False.

[serhiy-storchaka]

cc @ezio-melotti

This is a bug. According to specification, ampersand is appended to the temporary buffer and then flushed together with other appended characters if the name is not found in the named character references table.

https://html.spec.whatwg.org/multipage/parsing.html#character-reference-state
https://html.spec.whatwg.org/multipage/parsing.html#named-character-reference-state

[zangjiucheng]

Hello everyone,

I’ve been investigating this issue and stepping through the code. It seems the core problem occurs in this section:

cpython/Lib/html/parser.py

Lines 323 to 331 in d23e8db

	if match:
	# match.group() will contain at least 2 chars
	if end and match.group() == rawdata[i:]:
	k = match.end()
	if k <= i:
	k = n
	i = self.updatepos(i, i + 1)
	# incomplete
	break

At this point, the parser treats the input as an incomplete entity, while also reaching the end of the data stream. Because of that, the call to updatepos(i, i + 1) advances the index by one and effectively “eats” the & character before breaking out of the loop.

I propose to simply break out of the loop in this case—since we’re already at EOF—and let the final flush emit the entire literal string (e.g., &A). I might be missing some context, but I don’t see a clear reason why we should advance the position here before breaking.

if match:
-    # match.group() will contain at least 2 chars
-    if end and match.group() == rawdata[i:]:
-        k = match.end()
-        if k <= i:
-            k = n
-        i = self.updatepos(i, i + 1)
    # incomplete
    break

Alternatively, another solution that could consume the entire match is:

if match:
    if end and match.group() == rawdata[i:]:
        data = match.group()
        self.handle_data(data)
        i = self.updatepos(i, i + len(data))
        continue
    break

Reference commit: 73a4359

[serhiy-storchaka]

Yes, this is the right place, but solution should be different. The behavior for unclosed character reference followed by EOF and followed by some character should be consistent -- call handle_entityref() or handle_charref(), not handle_data(), if there is at least one character of the name or a digit). So, the user can resolve character references themselves. The issue affects also numerical character references.

There is still other unresolved issue. &gtundefined is parsed as characters >undefined with convert_charrefs=True, but you will get an entityref gtundefined with convert_charrefs=False -- the same as for &gtundefined;. So you cannot distinguish &gtundefined from &gtundefined; with convert_charrefs=False.