str.splitlines splitting on non-\r\n characters

[scharron]

BPO	22232
Nosy	@malemburg, @warsaw, @nascheme, @terryjreedy, @gpshead, @vstinner, @jwilk, @ezio-melotti, @stevendaprano, @bitdancer, @vadmium, @serhiy-storchaka
Files	cpython3.5_splitlines.diff cpython3.5_splitlines.diff cpython2.7_splitlines.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2014-08-20.10:01:51.226>
labels = ['type-bug', 'docs']
title = 'str.splitlines splitting on non-\\r\\n characters'
updated_at = <Date 2018-10-07.20:48:13.606>
user = 'https://bugs.python.org/scharron'

bugs.python.org fields:

activity = <Date 2018-10-07.20:48:13.606>
actor = 'nascheme'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation']
creation = <Date 2014-08-20.10:01:51.226>
creator = 'scharron'
dependencies = []
files = ['43071', '43072', '43075']
hgrepos = []
issue_num = 22232
keywords = ['patch']
message_count = 34.0
messages = ['225561', '225564', '225705', '225709', '225723', '225736', '225747', '225748', '225755', '225758', '225766', '225767', '225769', '225873', '225874', '230138', '230143', '230144', '230152', '246547', '246570', '266781', '266782', '266785', '266789', '266800', '327105', '327106', '327113', '327137', '327141', '327162', '327269', '327308']
nosy_count = 16.0
nosy_names = ['lemburg', 'barry', 'nascheme', 'terry.reedy', 'gregory.p.smith', 'vstinner', 'jwilk', 'ezio.melotti', 'steven.daprano', 'r.david.murray', 'docs@python', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'scharron', 'Alexander Schrijver']
pr_nums = []
priority = 'normal'
resolution = None
stage = 'needs patch'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue22232'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

Linked PRs

• gh-66428 Stop including all bidirectional "B" characters in line breakers #132369

[scharron]

According to the documentation, str.splitlines uses the universal newlines to split lines.
The documentation says it's all about \r, \n, and \r\n (https://docs.python.org/3.5/glossary.html#term-universal-newlines)

However, it's also splitting on other characters. Reading the code, it seems the list of characters is from Objects/unicodeobject.c , in _PyUnicode_Init, the linebreak array.
When testing any of these characters, it splits the string.

Other libraries are using str.splitlines assuming it only breaks on these \r and \n characters. This is the case of email.feedparser for instance, used by http.client to parse headers. These HTTP headers should be separated by CLRF as specified by http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.

Either the documentation should state that splitlines splits on other characters or it should stick to the documentation and split only on \r and \n characters.

If it splits on other characters, the list could be improved, as the unicode reference lists the mandatory characters for line breaking : http://www.unicode.org/reports/tr14/tr14-32.html#BK

[scharron]

For an example of a serious bug caused by this, see http://bugs.python.org/issue22233

[terryjreedy]

Objects/unicodeobject.c linebreak is at 266. With 3.4.1:
>>> 'a\x0ab\x0bc\x0cd\x0d1c\x1c1d\x1d1e\x1e'.splitlines()
['a', 'b', 'c', 'd', '1c', '1d', '1e']
\x0a == \n, \x0d == \r

The \r\n pair is a special case, as promised, but other pairs are not.
>>> 'a\r\nb'.splitlines()
['a', 'b']
>>> 'a\x0b\nb'.splitlines()
['a', '', 'b']

[scharron]

It's also at line bpo-14941 for unicode strings if I understand correctly

With 3.4.0:

>>> "a\x85b\x1ec".splitlines()
['a', 'b', 'c']

[bitdancer]

See bpo-7643 for some technical background. There are some other interesting issues to read if you seach the tracker for 'splitlines unicode', one of which is an open doc issue. Clearly the docs about this are inadequate.

Basically, though, I think you are correct. email should not be using splitlines(). It was more or less correct when email was splitting binary data, but even then it wasn't exactly correct per the letter of RFC.

Unfortunately not using splitlines has some performance implications...but then again we haven't done any sort of performance improvement pass on the new email code, so it may well be marginal in the overall scheme of things.

[scharron]

This is a known issue, and will be resolved by improving documentation, I'm closing this bug

Thanks !

[bitdancer]

OK, we'll use bpo-22232 to resolve the issue of email using splitlines.

[serhiy-storchaka]

May be add a parameter to str.splitlines() which will switch behavior to split on '\n', '\r' and '\r\n' only?

[terryjreedy]

Unless there is already another issue for improving the doc, this should at least be left open as a doc issue.

But I had the same thought as Serhiy, that we should at least optionally make the current doc correct. Two possibilities:

newlines=False  If true, only split on \r, \n, \r\n; otherwise split on all latin-1 linebreak characters -- <list>.  {This is rather awkward.}

linebreak=True  If true, split on all latin-1 linebreak characters <list>; otherwise only split on \r, \n, \r\n.  {Better, to me}

Changing both code and doc, at least in 3.5, says that both are wrong. If we agree on this, there is still the awkward issue of what to do for 3.4. Just change the doc? Then email must do something different in 3.4 to work around the code behavior. I think this may warrant a pydev discussion.

Another issue is whether latin-1 linebreaks are privileged. Why not implement the full unicode linebreak algorithm.

An additional complication is that in 2.x, .splitlines acts as advertised.

>>> 'a\x0ab\x0bc\x0cd\x0dda\x0d\x0a1c\x1c1d\x1d1e\x1e85\x85end'.splitlines()
['a', 'b\x0bc\x0cd', 'da', '1c\x1c1d\x1d1e\x1e85\x85end']