imaplib truncates some untagged responses

[rafales]

BPO	21815
Nosy	@warsaw, @ezio-melotti, @bitdancer, @berkerpeksag, @soltysh
Files	imaplib_log.txt imap_regex.patch imaplib_log_with_patch.txt test_imaplib.patch imap_doc.patch imap_regex.patch imaplib_bracket_fix.patch imaplib_after_review.patch imaplib_after_silentghost_review.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-01-02.22:24:50.004>
created_at = <Date 2014-06-20.14:39:56.350>
labels = ['type-bug', 'library', 'expert-email']
title = 'imaplib truncates some untagged responses'
updated_at = <Date 2016-01-02.22:26:16.496>
user = 'https://bugs.python.org/rafales'

bugs.python.org fields:

activity = <Date 2016-01-02.22:26:16.496>
actor = 'python-dev'
assignee = 'none'
closed = True
closed_date = <Date 2016-01-02.22:24:50.004>
closer = 'r.david.murray'
components = ['Library (Lib)', 'email']
creation = <Date 2014-06-20.14:39:56.350>
creator = 'rafales'
dependencies = []
files = ['35706', '35962', '35963', '35977', '35978', '35979', '37245', '41366', '41395']
hgrepos = []
issue_num = 21815
keywords = ['patch']
message_count = 39.0
messages = ['221091', '221092', '223070', '223163', '223164', '223168', '223170', '223172', '223204', '223207', '223239', '223254', '223265', '223297', '223302', '223303', '225322', '230467', '230517', '231516', '240657', '240701', '240712', '240792', '256077', '256121', '256126', '256534', '256599', '256752', '256879', '256885', '256902', '257172', '257195', '257241', '257372', '257374', '257376']
nosy_count = 9.0
nosy_names = ['barry', 'ezio.melotti', 'r.david.murray', 'jesstess', 'python-dev', 'berker.peksag', 'maciej.szulik', 'Lita.Cho', 'rafales']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue21815'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[rafales]

The regexp in Response_code checks for the existence of [] characters. The problem is that the strings may contain [] characters. I attach debug log which shows the data received from server and what the python extracted from it.

You can see that the PERNANENTFLAGS is truncated and everything from the ] character is lost:

(\Answered \Flagged \Draft \Deleted \Seen OIB-Seen-OIB/Social Networking $Phishing $Forwarded OIB-Seen-OIB/Home OIB-Seen-OIB/Shopping OIB-Seen-INBOX OIB-Seen-OIB/Business OIB-Seen-OIB/Entertainment $NotJunk $NotPhishing $Junk OIB-Seen-[Gmail

[rafales]

Oops, forgot the file.

[LitaCho]

I was reading the RFC spec, and it looks like it doesn't specificy '[' and ']' are not allowed in the PERNANENTFLAGS names.

I can try to add a fix for this. But if anyone else knows if you are not allowed to have '[' or ']' in flag names, please let me know.

[LitaCho]

I have a patch for this. With my patch, the debug output is fixed.

[LitaCho]

Here is a log of the output.

[jesstess]

Thanks for the patch, Lita! Lita: I have some patch feedback for you at the end of this message.

--

Lita and I talked about this a bit via email; here's some additional context from that conversation:

There were a couple of questions to answer before working on a patch:

1. Are brackets allowed in flag names according to the RFC?
2. If brackets aren't allowed, do other popular IMAP services permit them anyway? If so, we should consider allowing them even though this deviates from the RFC.

To answer (1), my read of http://tools.ietf.org/html/rfc3501 is as follows:

a. Look at http://tools.ietf.org/html/rfc3501, search for PERMANENTFLAGS definition. Note use of "flag-perm" variable.

b. Search for "flag-perm". Find it on page 85, in section 9, Formal Syntax. This is describing the formal syntax in Backus-Naur Form (BNF).

c. flag-perm = flag / "\*"

d. flag = "\Answered" / "\Flagged" / "\Deleted" /
"\Seen" / "\Draft" / flag-keyword / flag-extension
; Does not include "\Recent"

e. flag-keyword = atom

f. atom = 1*ATOM-CHAR

g. ATOM-CHAR = <any CHAR except atom-specials>

h. atom-specials = "(" / ")" / "{" / SP / CTL / list-wildcards /
quoted-specials / resp-specials

i. resp-specials = "]"

My reading of this series of definitions is that "]" is not allowed in flag names.

To answer (2), Lita ran some tests against GMail's IMAP service and found that it does let you create flags containing "]".

--

Patch feedback:

1. Can you attach the script you used to probe GMail's IMAP service for what flag names were permitted?

2. Since this patch causes us to violate the RFC (but with good reason!), can you add a comment above the regex noting the violation and stating what characters we allow for flags and why?

3. This change need tests.

[LitaCho]

Yes! I agree, this change will need tests. I will start working on creating those now.

Here is test I did to create a flag with brackets.

import imaplib

mail = imaplib.IMAP4_SSL('imap.gmail.com')
mail.login('username@gmail.com', 'password') # Enter your login here
mail.select('test') # Mailbox selection. I have a test inbox with 6
                    # messages in it.
code, [msg_ids] = mail.search(None, 'ALL')
first_id = msg_ids.split()[0]
mail.store(first_id, "+FLAGS", "[test]")
typ, response = mail.fetch(first_id, '(FLAGS)')
print("Flags: %s" % response)
mail.close()
mail.logout()

[LitaCho]

Here is the code in order to see the bug.

imaplib.Debug = 5
mail = imaplib.IMAP4_SSL('imap.gmail.com')
mail.login(username, password) # Enter your login here
mail.select('test')

[bitdancer]

Just to make sure I understand: the issue is that gmail may produce flags with [] in them, and imaplib currently fails to process such flags when it receives them from gmail?

In principle I think we would not want to allow imaplib to be used to create such flags unless the user specifies some sort of "I want to violate the RFC" flag (which they might want to do, for example, to run tests against gmail :) But currently it looks like it can? (I haven't looked at this in enough detail to be sure.) If that's true we probably have to continue to allow it for backward compatibility reasons, but we should document the RFC violation and possible consequences (an IMAP server rejecting such flags).

[rafales]

Yeah, basically. The flags with [] characters are added to gmail (in my
case) by OtherInbox's Organizer app. I contacted them but haven't heard
back yet.

On Wed, Jul 16, 2014 at 4:07 PM, R. David Murray <report@bugs.python.org>
wrote:

R. David Murray added the comment:

Just to make sure I understand: the issue is that gmail may produce flags
with [] in them, and imaplib currently fails to process such flags when it
receives them from gmail?

In principle I think we would not want to allow imaplib to be used to
create such flags unless the user specifies some sort of "I want to violate
the RFC" flag (which they might want to do, for example, to run tests
against gmail :) But currently it looks like it can? (I haven't looked at
this in enough detail to be sure.) If that's true we probably have to
continue to allow it for backward compatibility reasons, but we should
document the RFC violation and possible consequences (an IMAP server
rejecting such flags).

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue21815\>

---

[LitaCho]

R. David Murray added the comment:

Just to make sure I understand: the issue is that gmail may produce flags
with [] in them, and imaplib currently fails to process such flags when it
receives them from gmail?

This is correct. Gmail allows you to create flags with [], and the
Response_code regex doesn't process them properly.

In principle I think we would not want to allow imaplib to be used to
create such flags unless the user specifies some sort of "I want to violate
the RFC" flag (which they might want to do, for example, to run tests
against gmail :) But currently it looks like it can? (I haven't looked at
this in enough detail to be sure.) If that's true we probably have to
continue to allow it for backward compatibility reasons, but we should
document the RFC violation and possible consequences (an IMAP server
rejecting such flags).

Yes, currently we can. I've posted the code in order to do this. This is
basically the result.

>>> first_id = msg_ids.split()[0]
>>> mail.store(first_id, "+FLAGS", "[test]")
('OK', [b'1 (FLAGS (\\Seen Answered [test] NotJunk $NotJunk [Brackets]
[testing2]))'])

However, I would think it would be the server's job to uphold this rule,
not the library. The server should return with a BAD response, but right
now, Gmail allows you to do this.

Should we throw a warning in the "store" method? Otherwise, I can update
the documenation in the "store" method stating that having '[]' is allowed
but violates the RFC protocol.

----------

---

Python tracker <report@bugs.python.org>
<http://bugs.python.org/issue21815\>

---

[bitdancer]

Postel's Law says: be conservative in what you do, be liberal in what you accept from others. So the client should not violate the RFC when sending data to the server. The server, on the other hand, by that law could accept "dirty" data if it can do something reasonable with it...except that in general the IMAP RFC in particular says "don't do that". IMAP servers are supposed to be IMAP RFC sticklers, even when accepting input. So gmail is broken for some definition of broken, and so is imaplib.

Since the code has been this way for a long time, and gmail does in fact accept it, I think a doc warning about []s violating the RFC should be sufficient; a warning would probably just be annoying to little real benefit.