Test failures with Python 3.14.0b3

After upgrading Python 3.14.0b2 to Python 3.14.0b3, I'm seeing the following test failures:

$ tox -e py314
py314: recreate env because python changed version_info=[3, 14, 0, 'beta', 2]->[3, 14, 0, 'beta', 3]
py314: remove tox env folder /tmp/markdown/.tox/py314
py314: install_deps> python -I -m pip install pytidylib
.pkg: recreate env because python changed version_info=[3, 14, 0, 'beta', 2]->[3, 14, 0, 'beta', 3]
.pkg: remove tox env folder /tmp/markdown/.tox/.pkg
.pkg: install_requires> python -I -m pip install 'setuptools>=77.0'
.pkg: _optional_hooks> python /usr/lib/python3.14/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: get_requires_for_build_sdist> python /usr/lib/python3.14/site-packages/pyproject_api/_backend.py True setuptools.build_meta
.pkg: build_sdist> python /usr/lib/python3.14/site-packages/pyproject_api/_backend.py True setuptools.build_meta
py314: install_package_deps> python -I -m pip install coverage 'importlib-metadata>=4.4; python_version < "3.10"' pyyaml
py314: install_package> python -I -m pip install --force-reinstall --no-deps /tmp/markdown/.tox/.tmp/package/6/markdown-3.8.1.tar.gz
py314: commands[0]> coverage run --source=markdown -m unittest discover /tmp/markdown/tests
......................../tmp/markdown/markdown/__main__.py:103: DeprecationWarning: codecs.open() is deprecated. Use open() instead.
  with codecs.open(
................../tmp/markdown/markdown/core.py:420: DeprecationWarning: codecs.open() is deprecated. Use open() instead.
  input_file = codecs.open(input, mode="r", encoding=encoding)
/tmp/markdown/markdown/core.py:436: DeprecationWarning: codecs.open() is deprecated. Use open() instead.
  output_file = codecs.open(output, "w",
......................................................................................s...................................................................................sssssssssssssssssssssssssssssssssssssssssssssss...........................................................................ss...........................................................................................F.................................F.......................................F.....................ss...........................................................................................................................................................................................F.................................F.......................................F.............................................................................................................................................................................................................................................................................
======================================================================
FAIL: test_not_actually_cdata (test_syntax.blocks.test_html_blocks.TestHTMLBlocks.test_not_actually_cdata)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/markdown/tests/test_syntax/blocks/test_html_blocks.py", line 1280, in test_not_actually_cdata
    self.assertMarkdownRenders(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        '<![',
        ^^^^^^
        '<p>&lt;![</p>'
        ^^^^^^^^^^^^^^^
    )
    ^
  File "/tmp/markdown/markdown/test_tools.py", line 77, in assertMarkdownRenders
    self.assertMultiLineEqual(output, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: '<!--[\n\n-->' != '<p>&lt;![</p>'
+ <p>&lt;![</p>
- <!--[
- 
- -->


======================================================================
FAIL: test_raw_missing_close_bracket (test_syntax.blocks.test_html_blocks.TestHTMLBlocks.test_raw_missing_close_bracket)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/markdown/tests/test_syntax/blocks/test_html_blocks.py", line 662, in test_raw_missing_close_bracket
    self.assertMarkdownRenders(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        '<foo',
        ^^^^^^^
        '<p>&lt;foo</p>'
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/tmp/markdown/markdown/test_tools.py", line 77, in assertMarkdownRenders
    self.assertMultiLineEqual(output, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: '' != '<p>&lt;foo</p>'
+ <p>&lt;foo</p>


======================================================================
FAIL: test_unclosed_comment_ (test_syntax.blocks.test_html_blocks.TestHTMLBlocks.test_unclosed_comment_)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/markdown/tests/test_syntax/blocks/test_html_blocks.py", line 1022, in test_unclosed_comment_
    self.assertMarkdownRenders(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.dedent(
        ^^^^^^^^^^^^
    ...<11 lines>...
        )
        ^
    )
    ^
  File "/tmp/markdown/markdown/test_tools.py", line 77, in assertMarkdownRenders
    self.assertMultiLineEqual(output, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: '<!-- unclosed comment\n\n*not* a comment\n\n-->' != '<p>&lt;!-- unclosed comment</p>\n<p><em>not</em> a comment</p>'
- <!-- unclosed comment
+ <p>&lt;!-- unclosed comment</p>
?  ++++++                    ++++
+ <p><em>not</em> a comment</p>
- 
- *not* a comment
- 
- -->


======================================================================
FAIL: test_not_actually_cdata (test_syntax.extensions.test_md_in_html.TestDefaultwMdInHTML.test_not_actually_cdata)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/markdown/tests/test_syntax/blocks/test_html_blocks.py", line 1280, in test_not_actually_cdata
    self.assertMarkdownRenders(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        '<![',
        ^^^^^^
        '<p>&lt;![</p>'
        ^^^^^^^^^^^^^^^
    )
    ^
  File "/tmp/markdown/markdown/test_tools.py", line 77, in assertMarkdownRenders
    self.assertMultiLineEqual(output, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: '<!--[\n\n-->' != '<p>&lt;![</p>'
+ <p>&lt;![</p>
- <!--[
- 
- -->


======================================================================
FAIL: test_raw_missing_close_bracket (test_syntax.extensions.test_md_in_html.TestDefaultwMdInHTML.test_raw_missing_close_bracket)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/markdown/tests/test_syntax/blocks/test_html_blocks.py", line 662, in test_raw_missing_close_bracket
    self.assertMarkdownRenders(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        '<foo',
        ^^^^^^^
        '<p>&lt;foo</p>'
        ^^^^^^^^^^^^^^^^
    )
    ^
  File "/tmp/markdown/markdown/test_tools.py", line 77, in assertMarkdownRenders
    self.assertMultiLineEqual(output, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: '' != '<p>&lt;foo</p>'
+ <p>&lt;foo</p>


======================================================================
FAIL: test_unclosed_comment_ (test_syntax.extensions.test_md_in_html.TestDefaultwMdInHTML.test_unclosed_comment_)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/markdown/tests/test_syntax/blocks/test_html_blocks.py", line 1022, in test_unclosed_comment_
    self.assertMarkdownRenders(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self.dedent(
        ^^^^^^^^^^^^
    ...<11 lines>...
        )
        ^
    )
    ^
  File "/tmp/markdown/markdown/test_tools.py", line 77, in assertMarkdownRenders
    self.assertMultiLineEqual(output, expected)
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
AssertionError: '<!-- unclosed comment\n\n*not* a comment\n\n-->' != '<p>&lt;!-- unclosed comment</p>\n<p><em>not</em> a comment</p>'
- <!-- unclosed comment
+ <p>&lt;!-- unclosed comment</p>
?  ++++++                    ++++
+ <p><em>not</em> a comment</p>
- 
- *not* a comment
- 
- -->


----------------------------------------------------------------------
Ran 1056 tests in 1.743s

FAILED (failures=6, skipped=52)
py314: exit 1 (2.02 seconds) /tmp/markdown> coverage run --source=markdown -m unittest discover /tmp/markdown/tests pid=3540134
  py314: FAIL code 1 (8.97=setup[6.96]+cmd[2.02] seconds)
  evaluation failed :( (9.06 seconds)

This may be a duplicate of: #1534

Oh, you seem to be using the latest release (that was about 3 hours ago: 3.8.1).

Yes, sorry for not mentioning it. I'm so used to always retesting latest git that I forget to mention it.

Also it's definitely related to 3.14.0b3. The 3.8 release also fails like this, and both pass with 3.14.0b2. I just hope this isn't another regression in CPython.

These differences bother me ☹. They will definitely require further investigation.

I have confirmed these failures locally.

It seems the parser is now auto closing unclosed comments and then dropping content like <foo if not completed properly. I'm not sure how to fix it yet, but that seems to be what is happening at least.

The <foo case seems easy to handle in a backwards compatible way. I'm not sure about the autocompletion of comments yet.

I'm guessing this is python/cpython#135464. Given it's a security fix, it will probably be backported to all older Python versions too.

And confirmed. Tests pass prior to that change, fail with it.

So far, I can fix all cases except <!-- unclosed comment. It seems that right now, if the comment is unclosed, it consumes everything until the end.

Looks like this is getting backported to Python 3.9, 3.10, 3.11, 3.12, 3.13, and 3.14 so we need to get this fix out soon.

Apparently, Python has actually been applying fixes to the HTML parser recently, This is the third I'm aware of in the last few months. It took them 5 years to finally merge my pull request (see gh-86155. Maybe a new dev is actually interested in working on the lib!?

So far, I can fix all cases except <!-- unclosed comment. It seems that right now, if the comment is unclosed, it consumes everything until the end.

I was going to say that that behavior actually makes more sense as it matches how browsers parse HTML. Then I checked the tests and see this comment (at test_html_blocks.py#L1018-L1020):

    # Note: This is a change in behavior. Previously, Python-Markdown interpreted this in the same manner
    # as browsers and all text after the opening comment tag was considered to be in a comment. However,
    # that did not match the reference implementation. The new behavior does.

🤷‍♂

Fixes for Python 3.14 and fixes for deprecations

Created a draft PR. I don't have a solution for the comments consuming until they are closed or hit the end of the file. I'll need time to investigate that further.

Fixes for Python 3.14 and fixes for deprecations

I have backward-compatible fixes for all issues in Python 3.14. We can, of course, decide not to do some of them if we want to accept any of the breaking changes. I admit that some fixes are more hacky than others. At the very least, this exposes exactly what the issues are and can help us make a more informed decision (pros vs cons) of what it would take to address all the issues.

Fixes for Python 3.14

Thank you for the prompt fix!

Navigation Menu

Repository navigation

Test failures with Python 3.14.0b3 #1537

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Test failures with Python 3.14.0b3 #1537

Description

Activity

facelessuser commented on Jun 18, 2025

facelessuser commented on Jun 18, 2025

mgorny commented on Jun 18, 2025

mgorny commented on Jun 18, 2025

facelessuser commented on Jun 18, 2025

facelessuser commented on Jun 18, 2025

facelessuser commented on Jun 18, 2025

facelessuser commented on Jun 18, 2025

mgorny commented on Jun 18, 2025

mgorny commented on Jun 18, 2025

facelessuser commented on Jun 18, 2025

waylan commented on Jun 18, 2025

waylan commented on Jun 18, 2025

facelessuser commented on Jun 18, 2025

facelessuser commented on Jun 18, 2025

mgorny commented on Jun 19, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions