Email Parser use 100% CPU

[jaderfabiano]

BPO	21448
Nosy	@warsaw, @rhettinger, @pitrou, @tiran, @bitdancer, @serhiy-storchaka
Files	email_parser_long_lines.patch fix_email_parse.diff: Clean-up code in push() fix_email_parse2.diff: Revise patch to add splitlines. test_parser.diff: Extra test fix_prepending2.diff: Speed-up insertion by using a deque email_parser_long_lines_2.patch: Raymond's patch + tests

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-05-23.00:24:13.457>
created_at = <Date 2014-05-06.19:20:13.387>
labels = ['expert-email', 'performance']
title = 'Email Parser use 100% CPU'
updated_at = <Date 2015-05-23.00:24:13.456>
user = 'https://bugs.python.org/jaderfabiano'

bugs.python.org fields:

activity = <Date 2015-05-23.00:24:13.456>
actor = 'rhettinger'
assignee = 'none'
closed = True
closed_date = <Date 2015-05-23.00:24:13.457>
closer = 'rhettinger'
components = ['email']
creation = <Date 2014-05-06.19:20:13.387>
creator = 'jader.fabiano'
dependencies = []
files = ['36210', '36216', '36230', '36231', '36233', '36334']
hgrepos = []
issue_num = 21448
keywords = ['patch']
message_count = 30.0
messages = ['218008', '218010', '218011', '218012', '218013', '218014', '218015', '218016', '218123', '224516', '224562', '224591', '224610', '224623', '224624', '224631', '224647', '224649', '224667', '224863', '224866', '224894', '224910', '225155', '225158', '225228', '225229', '225233', '240670', '243873']
nosy_count = 9.0
nosy_names = ['barry', 'rhettinger', 'pitrou', 'christian.heimes', 'r.david.murray', 'tshepang', 'python-dev', 'serhiy.storchaka', 'jader.fabiano']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue21448'
versions = ['Python 3.5']

[jaderfabiano]

Use email.parser to catch the mime's header,when a mime has attachments the process is consuming 100% of the CPU. And It can take until four minutes to finish the header parser.

[bitdancer]

Can you provide more details on how to reproduce the problem, please? For example, a sample message and the sequence of python calls you use to parse it.

[jaderfabiano]

I am openning a file and I am passing the File Descriptor to this function
Parse().parse( fp ):
This file has two attachments
Example:
self.fileDescriptor( file, 'rb')
headers = Parser().parse(self.fileDescriptor )
#Here the process starts to consume 100% of the CPU and It takes around
four minutes to go the next line.
print 'Headers OK'

The File's size is 12M

Thanks.

2014-05-06 16:31 GMT-03:00 R. David Murray <report@bugs.python.org>:

R. David Murray added the comment:

Can you provide more details on how to reproduce the problem, please? For
example, a sample message and the sequence of python calls you use to parse
it.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue21448\>

---

[jaderfabiano]

Sorry!
Correct line
self.fileDescriptor = open( file, 'rb')

2014-05-06 16:51 GMT-03:00 jader fabiano <report@bugs.python.org>:

jader fabiano added the comment:

I am openning a file and I am passing the File Descriptor to this function
Parse().parse( fp ):
This file has two attachments
Example:
self.fileDescriptor( file, 'rb')
headers = Parser().parse(self.fileDescriptor )
#Here the process starts to consume 100% of the CPU and It takes around
four minutes to go the next line.
print 'Headers OK'

The File's size is 12M

Thanks.

2014-05-06 16:31 GMT-03:00 R. David Murray <report@bugs.python.org>:

>
> R. David Murray added the comment:
>
> Can you provide more details on how to reproduce the problem, please?
For
> example, a sample message and the sequence of python calls you use to
parse
> it.
>
> ----------
>
> _______________________________________
> Python tracker <report@bugs.python.org>
> <http://bugs.python.org/issue21448\>
> _______________________________________
>

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue21448\>

---

[bitdancer]

We'll need the data file as well. This is going to be a data-dependent issue. With a 12MB body, I'm guessing there's some decoding pathology involved, which may or may not have been already fixed in python3.

To confirm this you could use HeaderParser instead of Parser, which won't try to decode the body.

[jaderfabiano]

No, The file has 12Mb, because It has attachments. I am going to show an
example.

You can use a file thus:

Date: Tue, May 10:27:17 6 -0300 (BRT)
From: email@email.com.br
MIME-Version: 1.0
To: example@example.com
Subject:example

Content-Type: multipart/mixed; boundary=24f59adc-d522-11e3-a531-00265a0f1361

--24f59adc-d522-11e3-a531-00265a0f1361
Content-Type: multipart/alternative;
boundary=24f59a28-d522-11e3-a531-00265a0f1361

--24f59a28-d522-11e3-a531-00265a0f1361^M
Content-Type: text/html; charset="iso-8859-1" ^M
Content-Transfer-Encoding: 7bit

<br/><font color="bpo-00000" face="verdana" size="3">Test example</b>

--24f59a28-d522-11e3-a531-00265a0f1361--

--24f59adc-d522-11e3-a531-00265a0f1361
Content-Type: application/pdf; name=Example.pdf
Content-Disposition: attachment; filename=Example.pdf
Content-Transfer-Encoding: base64

attachment content in base64......

--24f59adc-d522-11e3-a531-00265a0f1361--

2014-05-06 17:03 GMT-03:00 R. David Murray <report@bugs.python.org>:

R. David Murray added the comment:

We'll need the data file as well. This is going to be a data-dependent
issue. With a 12MB body, I'm guessing there's some decoding pathology
involved, which may or may not have been already fixed in python3.

To confirm this you could use HeaderParser instead of Parser, which won't
try to decode the body.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue21448\>

---

[bitdancer]

Sorry, I was using RFC-speak. A message is divided into 'headers' and 'body', and all of the attachments are part of the body in RFC terms. But think of it as 'initial headers' and 'everything else'. Please either attach the full file, and/or try your test using HeaderParser and report the results.

However, it occurs to me that the attachments aren't decoded until you retrieve them, so whatever is going on it must be something other than a decoding issue. Nevertheless, Parser actually parses the whole message, attachments included, so we'll need the actual message in order to reproduce this (unless you can reproduce it with a smaller message).

[bitdancer]

Also to clarify: HeaderParser will *also* read the entire message, it just won't look for MIME attachments in the 'everything else', it will just treat the 'everything else' as arbitrary data and record it as the payload of the top level Message object.

[jaderfabiano]

Hi.
I undestood this problem that It was happening,
I was writting the mime wrong in the attachments. I read a file with size
4M and I've converted to Base64, so I've written in the mime the content.
But i wasn't put the lines with 76 ccharacters plus ""/r/n". I was writing
the every in the only line. I think this did the Email Parser uses 100% of
the CPU and It delay mora time.
I packed up and I was sending email very fast.

Thanks

2014-05-06 17:25 GMT-03:00 R. David Murray <report@bugs.python.org>:

R. David Murray added the comment:

Also to clarify: HeaderParser will *also* read the entire message, it just
won't look for MIME attachments in the 'everything else', it will just
treat the 'everything else' as arbitrary data and record it as the payload
of the top level Message object.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue21448\>

---

[serhiy-storchaka]

Therefore the bug is that email parser is dramatically slow for abnormal long lines. It has quadratic complexity from line size. Minimal example:

import email.parser
import time
data = 'From: example@example.com\n\n' + 'x' * 10000000
start = time.time()
email.parser.Parser().parsestr(data)
print(time.time() - start)

[serhiy-storchaka]

Parser reads from input file small chunks (8192 churacters) and feed FeedParser which pushes data into BufferedSubFile. In BufferedSubFile.push() chunks of incomplete data are accumulated in a buffer and repeatedly scanned for newlines. Every push() has linear complexity from the size of accumulated buffer, and total complexity is quadratic.

Here is a patch which fixes problem with parsing long lines. Feel free to add comments if they are needed (there is an abundance of comments in the module).

[rhettinger]

I think the push() code can be a little cleaner. Attaching a revised patch that simplifies push() a bit.