Bug: nl2br and toc extensions removes line that come immediatly after [toc]

[cbeauchesne]

Issue

On this specific markdown (pay attention to the unique line return after the toc marker)

[TOC]
where am I?

The nl2br and toc extensions used together remove the second line :

import markdown

parser = markdown.Markdown(extensions=['toc', 'nl2br'])

print(parser.convert("[TOC]\nWhere am I?",))

prints :

<div class="toc">
<ul></ul>
</div>

---

• Add another \n between the marker and the seconde line, and it's ok.
• Do not use nl2br, and it's also ok (in this case, the toc is not created, but I guess it's the expected behavior, as [toc] should be a block).

Expected behavior

From what I understand, and regarding previous versions (tested on Markdown==3.1.1), the result should be:

<p>[TOC]<br>
x</p>

Context

• Python 3.9.6 and Python 3.7.3
• Markdown==3.3.4
• both 5.10.25-linuxkit and MacOS

[facelessuser]

While this may be considered a bug, you really shouldn't make the [TOC] part of another paragraph. Having [TOC] in its own paragraph would resolve your issue, but, technically, I would consider this a bug.

[cbeauchesne]

Hi @facelessuser,

you really shouldn't make the [TOC] part of another paragraph

Yes I know 😅 , I expect this pattern not being populated as a toc.

I manage a wiki with more than 80,000 users and 1,000,000 documents, and almost all of them don't know markdown, and write it with a trial/error process. Any inconsistency in our markdown parser end up with terrible hacks in our base :/

A lead for the root cause: if I use TocTreeprocessor.iterparent and TocTreeprocessor.replace_marker from 3.3.1, I do not have the issue.

I'll try to dig it the next days.

[cbeauchesne]

A lead for the root cause: if I use TocTreeprocessor.iterparent and TocTreeprocessor.replace_marker from 3.3.1, I do not have the issue.

That's not true, sorry, it was a version I used locally.

---

So, the answer. On line 198: https://github.com/Python-Markdown/markdown/blob/master/markdown/extensions/toc.py#L195-L198

c is an element, and we check that c.text == marker.

But, with nl2br extension, it's possible to have children in c. In our case, it would be <p>[TOC]<br />Where am I?</p>. And as it comes from an extension, the br is a real Element, not a placeholder.

The issue is that, when an element is in that situation, c.text returns only the very first content, so [TOC]. And the extension entirely replace the content of the <p>.

To fix this, simply add a condition on len(c) == 0.

I'll propose a PR.

[waylan]

My first reaction was to say that this is user error and no bug exists. However, on closer inspection, I realized that there is data loss (the second line of the "paragraph" is not included in the output) and that is a bug.

[cbeauchesne]

Thanks @waylan !

Last question : do you know when you'll release the next version ?

And again, thank you for this great project !