email.parser.Parser hang

[Guido]

BPO	37461
Nosy	@warsaw, @vstinner, @larryhastings, @ned-deily, @alex, @bitdancer, @postmasters, @maxking, @eamanu, @miss-islington, @tirkarthi, @iritkatriel
PRs	[WIP] bpo-37461: Fix email.parser.Parse hang #14551 bpo-37461: Fix infinite loop in parsing of specially crafted email headers #14794 [3.7] bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794) #14816 [3.6] bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794) #14817 [3.8] bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794) #14818 [3.7] bpo-37461: Fix typo (inifite -> infinite) #15430 [3.6] bpo-37461: Fix typo (inifite -> infinite) #15432 [3.5] bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794) #15446
Files	reproducer.py reproducer2.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2020-10-19.21:45:01.609>
created_at = <Date 2019-07-01.03:21:13.231>
labels = ['type-security', '3.7', '3.8', 'expert-email', '3.9']
title = 'email.parser.Parser hang'
updated_at = <Date 2020-10-19.21:45:01.607>
user = 'https://bugs.python.org/Guido'

bugs.python.org fields:

activity = <Date 2020-10-19.21:45:01.607>
actor = 'barry'
assignee = 'none'
closed = True
closed_date = <Date 2020-10-19.21:45:01.609>
closer = 'barry'
components = ['email']
creation = <Date 2019-07-01.03:21:13.231>
creator = 'Guido'
dependencies = []
files = ['48483', '48484']
hgrepos = []
issue_num = 37461
keywords = ['patch', 'security_issue']
message_count = 19.0
messages = ['346953', '347014', '347108', '347263', '347953', '348060', '348064', '348070', '348071', '348072', '348850', '348867', '350344', '350345', '350346', '350348', '351289', '378396', '379027']
nosy_count = 14.0
nosy_names = ['barry', 'vstinner', 'larry', 'ned.deily', 'alex', 'r.david.murray', 'Nam.Nguyen', 'maxking', 'Guido', 'eamanu', 'miss-islington', 'xtreak', 'Marcin Niemira', 'iritkatriel']
pr_nums = ['14551', '14794', '14816', '14817', '14818', '15430', '15432', '15446']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'security'
url = 'https://bugs.python.org/issue37461'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7', 'Python 3.8', 'Python 3.9']

[Guido]

The following will hang, and consume a large amount of memory:

from email.parser import BytesParser, Parser
from email.policy import default
payload = "".join(chr(c) for c in [0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x78, 0x3b, 0x61, 0x72, 0x1b, 0x2a, 0x3d, 0x22, 0x73, 0x4f, 0x27, 0x23, 0x61, 0xff, 0xff, 0x27, 0x5c, 0x22])
Parser(policy=default).parsestr(payload)

[MarcinNiemira]

Sounds like there is an infinite loop here:

Pdb) 
> /usr/lib/python3.7/email/_header_value_parser.py(2370)get_parameter()
-> v.append(token)
(Pdb) 
> /usr/lib/python3.7/email/_header_value_parser.py(2365)get_parameter()
-> while value:

the v.append(token) is just growing with ValueTerminal(''), ValueTerminal(''), ValueTerminal('')...

I'd be happy to try to fix this.

[tirkarthi]

Since the parser could take user input this looks like a security issue to me along with high CPU usage. Feel free to remove the tag if it's not a security issue. Thanks.

[MarcinNiemira]

I'm terribly sorry, but I feel I won't be able to fix this issue. Sorry for fuss. Closing my PR, because it's broken.

[vstinner]

>>> bytes([0x43, 0x6f, 0x6e, 0x74, 0x65, 0x6e, 0x74, 0x2d, 0x54, 0x79, 0x70, 0x65, 0x3a, 0x78, 0x3b, 0x61, 0x72, 0x1b, 0x2a, 0x3d, 0x22, 0x73, 0x4f, 0x27, 0x23, 0x61, 0xff, 0xff, 0x27, 0x5c, 0x22])
b'Content-Type:x;ar\x1b*="sO\'#a\xff\xff\'\\"'

The following loop of Lib/email/_header_value_parser.py does never stop:

def get_parameter(value):
    """ attribute [section] ["*"] [CFWS] "=" value

The CFWS is implied by the RFC but not made explicit in the BNF.  This
simplified form of the BNF from the RFC is made to conform with the RFC BNF
through some extra checks.  We do it this way because it makes both error
recovery and working with the resulting parse tree easier.
"""
...
if remainder is not None:
    ...
    while value:
        ...

Attached reproducer.py is code from initial msg346953.

reproducer2.py simplify the input and calls directly get_parameter(). Simplified input string:

r*="'a'\"

[maxking]

I have proposed a PR for this: #14794

Reviews are welcome.

[Guido]

I used fuzzing to find this bug. After applying your patch, the infinite loop is gone and it cannot find any other bugs of this nature.

[warsaw]

New changeset a4a994b by Barry Warsaw (Abhilash Raj) in branch 'master':
bpo-37461: Fix infinite loop in parsing of specially crafted email headers (GH-14794)
a4a994b