tempfile.mkdtemp() doesn't work with bytes paths

[durin42]

BPO	24230
Nosy	@birkenfeld, @gpshead, @ncoghlan, @pitrou, @vstinner, @bitdancer, @vadmium, @serhiy-storchaka
Files	issue24230-gps01.patch issue24230-gps02.diff issue24230-gps03.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2015-05-22.23:21:09.422>
created_at = <Date 2015-05-18.17:04:53.985>
labels = ['type-feature', 'library']
title = "tempfile.mkdtemp() doesn't work with bytes paths"
updated_at = <Date 2016-03-15.11:46:23.756>
user = 'https://bugs.python.org/durin42'

bugs.python.org fields:

activity = <Date 2016-03-15.11:46:23.756>
actor = 'vstinner'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2015-05-22.23:21:09.422>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2015-05-18.17:04:53.985>
creator = 'durin42'
dependencies = []
files = ['39436', '39464', '39469']
hgrepos = []
issue_num = 24230
keywords = ['patch']
message_count = 23.0
messages = ['243498', '243500', '243502', '243505', '243507', '243508', '243509', '243552', '243643', '243648', '243649', '243654', '243672', '243674', '243675', '243809', '243821', '243847', '243865', '243866', '243876', '261810', '261811']
nosy_count = 11.0
nosy_names = ['georg.brandl', 'gregory.p.smith', 'ncoghlan', 'pitrou', 'vstinner', 'durin42', 'r.david.murray', 'python-dev', 'martin.panter', 'Matt.Mackall', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24230'
versions = ['Python 3.5']

[durin42]

Many things in the OS module work with bytestrings for paths in Python 3, but tempfile isn't so happy. It'd be very useful to be able to have the same bytes-in-bytes-out behavior in tempfile as for the os.path.* methods.

This is something we're working around in hg, but I figured I'd file a bug anyway in case we can improve the situation.

[bitdancer]

Can you explain what you are looking for in more detail? It isn't obvious to me what "bytes in bytes out" means in the context of the tempfile API.

[durin42]

Today we're doing something like this:

tmpdir = tempfile.mkdtemp('', 'hgtests.', d and d.decode('utf-8')).encode('utf-8')

but would rather have

tmpdir = tempfile.mkdtemp(b'', b'hgtests.', d)
# and now tmpdir is a bytestring containing the path to the tmpdir

basically, we want to maintain our pedantry around paths being sequences of bytes.

[MattMackall]

Another way of putting it is:

os.listdir(<bytes>) -> [<bytes>,...]
os.listdir(<unicode>) -> [<unicode>,...]

is the usual pattern, and tempfile isn't following it.

[bitdancer]

I understand now. The problem is that this would be a backward incompatible change.

[bitdancer]

I'm sorry, hit send before I finished thinking. You are saying bytes input are rejected, so yes this could be fixed.

[bitdancer]

Would you care to supply a patch? The beta deadline is this coming weekend.

[gpshead]

it seems a little messy to code this up the way Lib/tempfile.py is written but i'll take a stab at it tonight. Probably via parallel implementations of the methods internally rather than conditional logic throughout given how they work.

[gpshead]

Attached is a patch with implementation, tests and docs. I didn't have to duplicate too much thankfully. Just figure out where to put the type conversions. Review would be nice, but I'll err on the side of getting this in before beta 1 (May 24 per https://www.python.org/dev/peps/pep-0478/) so any corrections can be dealt with during the beta cycle.

Augie & Matt, can you confirm that this patch does what y'all need for Mercurial?

[vadmium]

Doing a quick review, expect a few comments on Rietveld

[serhiy-storchaka]

Added comments on Rietveld. But I'm not sure that this change is worth to do. Only low-level functions in os.path and os modules (and few other) support bytes names (and only bytes, not bytearray!). High level interfaces, such as in tarfile or zipfile, work only with str paths. The question is what level the tempfile module?

In any way Mercurial people need their own wrapper if they want to support Python <3.5.

[bitdancer]

We have a number of other places in the stdlib where bytes-in-bytes-out is observed, at various levels of abstraction. I think this is reasonable. To answer your direct question, I think tempfile is a convenience-and-do-it-right wrapper around what is really a pretty low level operation (creating a temporary file/directory securely).