'UnstructuredTokenList' object has no attribute '_fold_as_ew'

[touilleMan]

BPO	27240
Nosy	@warsaw, @bitdancer, @berkerpeksag, @touilleMan, @ArchibaldArt, @rslinckx, @vtemian
PRs	bpo-27240 Rewrite the email header folding algorithm. #3488 [3.6] bpo-27240 Rewrite the email header folding algorithm. (GH-3488) #4693
Files	test_mail_2.py test2.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-12-05.12:21:29.872>
created_at = <Date 2016-06-06.10:25:59.132>
labels = ['type-bug', '3.7', 'expert-email']
title = "'UnstructuredTokenList' object has no attribute '_fold_as_ew'"
updated_at = <Date 2018-11-17.14:54:32.694>
user = 'https://github.com/touilleMan'

bugs.python.org fields:

activity = <Date 2018-11-17.14:54:32.694>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2017-12-05.12:21:29.872>
closer = 'r.david.murray'
components = ['email']
creation = <Date 2016-06-06.10:25:59.132>
creator = 'touilleMan'
dependencies = []
files = ['43257', '46941']
hgrepos = []
issue_num = 27240
keywords = ['patch']
message_count = 11.0
messages = ['267525', '295751', '301456', '301481', '301796', '307528', '307530', '307642', '307659', '330016', '330030']
nosy_count = 8.0
nosy_names = ['barry', 'r.david.murray', 'berker.peksag', 'touilleMan', 'archibald', 'Zhang Wenbo', 'rslinckx', 'vtemian']
pr_nums = ['3488', '4693']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue27240'
versions = ['Python 3.6', 'Python 3.7']

[touilleMan]

The folding code for email headers crashes whenever it hits line _header_value_parser.py:432 given it try to call method _fold_as_ew.

(venv)➜  vigiechiro-api git:(in2p3) ✗ python test_mail.py  
Traceback (most recent call last):
  File "test_mail.py", line 36, in <module>
    mail.send(msg)
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 492, in send
    message.send(connection)
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 427, in send
    connection.send(self)
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 190, in send
    message.as_bytes() if PY3 else message.as_string(),
  File "/home/emmanuel/projects/vigiechiro-api/venv/lib/python3.5/site-packages/flask_mail.py", line 385, in as_bytes
    return self._message().as_bytes()
  File "/usr/lib/python3.5/email/message.py", line 179, in as_bytes
    g.flatten(self, unixfrom=unixfrom)
  File "/usr/lib/python3.5/email/generator.py", line 115, in flatten
    self._write(msg)
  File "/usr/lib/python3.5/email/generator.py", line 195, in _write
    self._write_headers(msg)
  File "/usr/lib/python3.5/email/generator.py", line 422, in _write_headers
    self._fp.write(self.policy.fold_binary(h, v))
  File "/usr/lib/python3.5/email/policy.py", line 190, in fold_binary
    folded = self._fold(name, value, refold_binary=self.cte_type=='7bit')
  File "/usr/lib/python3.5/email/policy.py", line 204, in _fold
    return self.header_factory(name, ''.join(lines)).fold(policy=self)
  File "/usr/lib/python3.5/email/headerregistry.py", line 255, in fold
    return header.fold(policy=policy)
  File "/usr/lib/python3.5/email/_header_value_parser.py", line 300, in fold
    self._fold(folded)
  File "/usr/lib/python3.5/email/_header_value_parser.py", line 1228, in _fold
    rest._fold(folded)
  File "/usr/lib/python3.5/email/_header_value_parser.py", line 432, in _fold
    part._fold_as_ew(folded)
AttributeError: 'UnstructuredTokenList' object has no attribute '_fold_as_ew'
> /usr/lib/python3.5/email/_header_value_parser.py(432)_fold()
-> part._fold_as_ew(folded)

Given folding depends a lot on the processed value, this bug only trigger in very specific encoded headers (I've try to reproduce this bug with various values but only the one which originally made me find the bug worked so far).

I've isolated the bug into a simple test case (see enclosed file).

It seems this part of the code has been rewritten between Python2 and 3 (the file _header_value_parser.py doesn't exists in Python2). So unsurprisingly the bug doesn't trigger in Python2.

Replacing part._fold_as_ew(folded) by part._fold(folded) seems to fix the bug, but given my poor knowledge of this code, this fix probably produce unwanted behaviors.

[ZhangWenbo]

I think I find a case which can reproduct this bug:
the UnstructuredTokenList has a ValueTerminal which is long enough(its encoded word is longer than policy.maxlen ) and not at the beginning.

For examples, the following subjects can cause this bug:

• 'Re: 我只是一个有那么一点点长的主题而已'
• '回复: 回复:【北京大学-2017年招生信息公告】'

So, the code can not fold a ValueTerminal correctly.
Seems the method _fold_as_ew was designed to do this, but it is missing.

[rslinckx]

It happens regularly for emails with long header lines for me too, which makes the python3 email module quite unusable.

A workaround which seems to help a bit is to set refold_source='none' on the email policy, although it seems to keep happening, less often.

Is the fix proposed by Emmanuel a proper fix ? I'm not knowledgeable enough on this subject...

[bitdancer]

I started rewriting the header folder in response to this bug, because the root problem is a bit deeper than just _fold_as_ew being missing. I will probably work on it some more this week, but if I don't think I'm going to get it finished I'll look at putting in a fix that doesn't address the other problems but does address this one.

[bitdancer]

I managed to finish the rewrite, and it does fix this issue. I'll probably post the PR tomorrow.

[bitdancer]

New changeset 85d5c18 by R. David Murray in branch 'master':
bpo-27240 Rewrite the email header folding algorithm. (bpo-3488)
85d5c18

[bitdancer]

New changeset a87ba60 by R. David Murray (Miss Islington (bot)) in branch '3.6':
bpo-27240 Rewrite the email header folding algorithm. (GH-3488) (bpo-4693)
a87ba60

[berkerpeksag]

All three examples in test2.py correctly folded (doesn't hit the ``except Exception as e:`` branch anymore) in current master. Can this issue be closed now?

[bitdancer]

Huh, I thought I had closed it.

[vtemian]

Is there any chance that this fix is gonna be ported to 3.5 as well?

[bitdancer]

No. 3.5 is in security-fix-only mode.