smtplib's SMTP.send_message behaves differently with from_addr and to_addrs

[thunderk]

BPO	32727
Nosy	@ned-deily, @bitdancer, @matrixise, @thunderk
PRs	bpo-32727: smtplib's SMTP.send_message behaves differently with from_addr and to_addrs #5451 [3.6] bpo-32727: smtplib's SMTP.send_message behaves differently with from_addr and to_addrs (GH-5451) #5455

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2018-01-31.21:05:31.138>
created_at = <Date 2018-01-30.15:10:32.950>
labels = ['3.7', '3.8', 'type-bug', 'library']
title = "smtplib's SMTP.send_message behaves differently with from_addr and to_addrs"
updated_at = <Date 2018-01-31.23:33:09.580>
user = 'https://github.com/thunderk'

bugs.python.org fields:

activity = <Date 2018-01-31.23:33:09.580>
actor = 'ned.deily'
assignee = 'none'
closed = True
closed_date = <Date 2018-01-31.21:05:31.138>
closer = 'r.david.murray'
components = ['Library (Lib)']
creation = <Date 2018-01-30.15:10:32.950>
creator = 'thunderk'
dependencies = []
files = []
hgrepos = []
issue_num = 32727
keywords = ['patch']
message_count = 17.0
messages = ['311275', '311276', '311278', '311279', '311280', '311282', '311284', '311286', '311296', '311297', '311298', '311301', '311302', '311306', '311370', '311374', '311385']
nosy_count = 4.0
nosy_names = ['ned.deily', 'r.david.murray', 'matrixise', 'thunderk']
pr_nums = ['5451', '5455']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue32727'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[thunderk]

Hi,

I noticed that SMTP.send_message, when getting the sender and recipients from the Message object, strips the name from recipients (to keep only the address), but not from the sender.

        if from_addr is None:
            # Prefer the sender field per RFC 2822:3.6.2.
            from_addr = (msg[header_prefix + 'Sender']
                           if (header_prefix + 'Sender') in msg
                           else msg[header_prefix + 'From'])
        if to_addrs is None:
            addr_fields = [f for f in (msg[header_prefix + 'To'],
                                       msg[header_prefix + 'Bcc'],
                                       msg[header_prefix + 'Cc'])
                           if f is not None]
            to_addrs = [a[1] for a in email.utils.getaddresses(addr_fields)]

There is an ugly side-effect to that (starting with Python 3.5) : if the sender name contains a non-ascii character, send_message will then require the SMTPUTF8 option from the SMTP server, and raise a SMTPNotSupportedError if unavailable. This is not wanted because the sender name is not actually sent to the SMTP server in the "MAIL FROM:" command (it is only sent in the MIME payload), so the SMTPUTF8 option should not be required based on it (it should only depend on the addresses).

[matrixise]

Do you have an example with a test?

Thank you

[bitdancer]

Yes, that looks like a bug. Should be a one line bug fix, plus tests and news item if someone wants to make a PR...

[thunderk]

Sure :

#coding: utf-8
import email.utils
from email.message import EmailMessage
from smtplib import SMTP
m = EmailMessage()
m['From'] = email.utils.formataddr(("Michaël", "michael@example.com"))
m['To'] = email.utils.formataddr(("René", "bounce@rodacom.fr"))
with SMTP('localhost') as smtp:
    smtp.set_debuglevel(2)
    smtp.send_message(m)
#END

On a server without SMTPUTF8, this outputs:

---

16:39:26.351302 send: 'ehlo localhost\r\n'
16:39:26.351391 reply: b'250-localhost\r\n'
16:39:26.351414 reply: b'250-PIPELINING\r\n'
16:39:26.351427 reply: b'250-SIZE 10240000\r\n'
16:39:26.351437 reply: b'250-VRFY\r\n'
16:39:26.351448 reply: b'250-ETRN\r\n'
16:39:26.351458 reply: b'250-ENHANCEDSTATUSCODES\r\n'
16:39:26.351468 reply: b'250-8BITMIME\r\n'
16:39:26.351477 reply: b'250 DSN\r\n'
16:39:26.351490 reply: retcode (250); Msg: b'localhost\nPIPELINING\nSIZE 10240000\nVRFY\nETRN\nENHANCEDSTATUSCODES\n8BITMIME\nDSN'
16:39:26.351832 send: 'QUIT\r\n'
16:39:26.351901 reply: b'221 2.0.0 Bye\r\n'
16:39:26.351923 reply: retcode (221); Msg: b'2.0.0 Bye'
Traceback (most recent call last):
  File "/usr/lib/python3.5/smtplib.py", line 943, in send_message
    ''.join([from_addr, *to_addrs]).encode('ascii')
UnicodeEncodeError: 'ascii' codec can't encode character '\xeb' in position 5: ordinal not in range(128)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "testsmtp.py", line 12, in <module>
    smtp.send_message(m)
  File "/usr/lib/python3.5/smtplib.py", line 947, in send_message
    "One or more source or delivery addresses require"
smtplib.SMTPNotSupportedError: One or more source or delivery addresses require internationalized email support, but the server does not advertise the required SMTPUTF8 capability

---

With removing the accented character in the sender name :
m['From'] = email.utils.formataddr(("Michael", "michael@example.com"))

It works :

---
16:42:49.882358 send: 'ehlo localhost\r\n'
16:42:49.882437 reply: b'250-localhost\r\n'
16:42:49.882460 reply: b'250-PIPELINING\r\n'
16:42:49.882474 reply: b'250-SIZE 10240000\r\n'
16:42:49.882486 reply: b'250-VRFY\r\n'
16:42:49.882498 reply: b'250-ETRN\r\n'
16:42:49.882509 reply: b'250-ENHANCEDSTATUSCODES\r\n'
16:42:49.882520 reply: b'250-8BITMIME\r\n'
16:42:49.882530 reply: b'250 DSN\r\n'
16:42:49.882543 reply: retcode (250); Msg: b'localhost\nPIPELINING\nSIZE 10240000\nVRFY\nETRN\nENHANCEDSTATUSCODES\n8BITMIME\nDSN'
16:42:49.883439 send: 'mail FROM:<michael@example.com> size=86\r\n'
16:42:49.886567 reply: b'250 2.1.0 Ok\r\n'
16:42:49.886603 reply: retcode (250); Msg: b'2.1.0 Ok'
16:42:49.886666 send: 'rcpt TO:<bounce@rodacom.fr>\r\n'
16:42:49.889642 reply: b'250 2.1.5 Ok\r\n'
16:42:49.889676 reply: retcode (250); Msg: b'2.1.5 Ok'
16:42:49.889694 send: 'data\r\n'
16:42:49.889769 reply: b'354 End data with <CR><LF>.<CR><LF>\r\n'
16:42:49.889827 reply: retcode (354); Msg: b'End data with <CR><LF>.<CR><LF>'
16:42:49.889843 data: (354, b'End data with <CR><LF>.<CR><LF>')
16:42:49.889939 send: b'From: Michael <michael@example.com>\r\nTo: =?utf-8?q?Ren=C3=A9?= <bounce@rodacom.fr>\r\n\r\n.\r\n'
16:42:49.892726 reply: b'250 2.0.0 Ok: queued as D92C540105\r\n'
16:42:49.892752 reply: retcode (250); Msg: b'2.0.0 Ok: queued as D92C540105'
16:42:49.892764 data: (250, b'2.0.0 Ok: queued as D92C540105')
16:42:49.892786 send: 'QUIT\r\n'
16:42:49.892862 reply: b'221 2.0.0 Bye\r\n'
16:42:49.892896 reply: retcode (221); Msg: b'2.0.0 Bye'
---

As you can see, the sender and recipients names are not sent in SMTP command, except in the MIME body, so they should not be used to determine whether SMTPUTF8 is required or not.

[thunderk]

Oups @r.david.murray, just saw I posted over your message, and removed you from the nozy list, sorry.

[matrixise]

@r.david.murray I started to work on a fix.

[matrixise]

@r.david.murray I am interested with your opinion about a test for this PR, if you have an idea, because all the tests pass on my laptop

[bitdancer]

Yes, you'll have to write a test. Add a test after test_send_unicode_without_SMTPUTF8, but put the unicode in the name field of the 'from' address and test that it is accepted. You should be able to figure out how to check for success from the other tests in that class (I hope ;)

[matrixise]

@r.david.murray I just updated the PR with a test, but I am not sure about it, could you give me your opinion because the current test suite works fine without my new test.

[bitdancer]

The current test suite had better work fine without your new test, otherwise your fix broke something :) I will take a look.

[matrixise]

yep, you can take a look but all the tests are green on Travis, maybe you will understand why I don't see the interest of my new test. you are the master for the email part ;-)