os.getcwd() hardcodes max path len

[pitrou]

BPO	9246
Nosy	@loewis, @terryjreedy, @pitrou, @vstinner, @giampaolo, @zware, @serhiy-storchaka, @worr
Files	bigpath.py os_getcwd_buffer-2.patch os_getcwd_maxpathlen.patch bigpath2.py max_getcwd.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-04-24.22:26:40.134>
created_at = <Date 2010-07-13.12:10:57.992>
labels = ['extension-modules', 'type-bug']
title = 'os.getcwd() hardcodes max path len'
updated_at = <Date 2015-04-24.22:26:40.133>
user = 'https://github.com/pitrou'

bugs.python.org fields:

activity = <Date 2015-04-24.22:26:40.133>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2015-04-24.22:26:40.134>
closer = 'vstinner'
components = ['Extension Modules']
creation = <Date 2010-07-13.12:10:57.992>
creator = 'pitrou'
dependencies = []
files = ['17983', '18225', '22394', '22395', '39199']
hgrepos = []
issue_num = 9246
keywords = ['patch']
message_count = 25.0
messages = ['110177', '110180', '110192', '110194', '110949', '111389', '111391', '111413', '111414', '111737', '111764', '111844', '111848', '138508', '138509', '138510', '138513', '138557', '185235', '185238', '240987', '241695', '241959', '241984', '241985']
nosy_count = 11.0
nosy_names = ['loewis', 'terry.reedy', 'pitrou', 'vstinner', 'giampaolo.rodola', 'boya', 'santoso.wijaya', 'python-dev', 'zach.ware', 'serhiy.storchaka', 'worr']
pr_nums = []
priority = 'high'
resolution = 'fixed'
stage = 'needs patch'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue9246'
versions = ['Python 3.5']

[pitrou]

In 2.x, os.getcwd() uses a dynamic allocation scheme to accomodate whatever buffer size the current path needs to be represented.

In 3.x, the max path length is hardcoded to 1026 bytes or characters, and an error is raised if the current path length is larger than that. Even on systems where MAX_PATH is 1024 (a common value), it is still valid to create paths larger than that (using e.g. os.mkdir()).

The attached script shows that os.getcwd() works with a 1032-character path in 2.x, but fails in 3.x.

[pitrou]

Even on systems where MAX_PATH is 1024 (a common value), it is still
valid to create paths larger than that (using e.g. os.mkdir()).

It seems I am mistaken on that. MAX_PATH is actually 4096 on the Linux system I am testing on. Calling getcwd() in a path longer than that fails with ENAMETOOLONG.

Still, 1026 shouldn't be the hard coded max length.

[skrah]

Just as a reminder: In 2.x, posix_getcwdu() also uses a buffer of size
1026.

[pitrou]

Just as a reminder: In 2.x, posix_getcwdu() also uses a buffer of size
1026.

I suppose the implementation was simply copied into py3k, then.
Still, it's not a very good idea and it will also be a regression when
porting scripts from 2.x to 3.x.

[skrah]

Closed bpo-6817 as a duplicate of this one. There are some patches in
that issue.

[terryjreedy]

On WinXP, 3.1, I get
...
mkdir: 242
Traceback (most recent call last):
  File "C:\Programs\Python31\misc\t1.py", line 14, in <module>
    os.mkdir(s)
WindowsError: [Error 206] The filename or extension is too long: 'C:\\Programs\\Python31\\misc\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab'

[skrah]

Terry J. Reedy <report@bugs.python.org> wrote:
> mkdir: 242
> Traceback (most recent call last):
>   File "C:\Programs\Python31\misc\t1.py", line 14, in <module>
>     os.mkdir(s)
> WindowsError: [Error 206] The filename or extension is too long: 'C:\\Programs\\Python31\\misc\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab\\aaaaab'

On Windows MAX_PATH seems to be 260 characters:

http://msdn.microsoft.com/en-us/library/aa365247(VS.85).aspx

[vstinner]

Patch based on Python 2 source code, but raises a MemoryError (instead of an OSError) on memory allocation failure.

With my patch, bigpath.py ends with "cwd: 1028 ...aab/aaaaab" with Python Python 3.2. Same result with Python 2.6. 1028 is bigger than 1026 (previous hardcoded max length in bytes including nul byte).

[vstinner]

I'm not sure that PyMem_Realloc(NULL, size) is always equivalent to PyMem_Malloc(size). And I don't really know why I'm using PyMem_* instead of malloc() / free() :-) I suppose that Python has a faster memory allocator, or that it has better checks when compiled with pydebug?

[pitrou]

It's not ok to call PyMem_* functions when the GIL is released. You should only release the GIL around the call to the system getcwd().

I suppose that Python has a faster memory allocator, or that it has
better checks when compiled with pydebug?

In this case it doesn't really make a difference, since all allocations larger than 256 bytes are delegated to the system allocator.

[vstinner]

New version of the patch avoiding PyMem_*() functions to avoid a possible race condition (issue with the GIL).

[vstinner]

Antoine asked me why not using a buffer of MAX_PATH+1 (instead of a dynamic buffer size). I don't know, I just copied/pasted the code from Python2. Extract of getcwd() manpage:

Note that on some systems, PATH_MAX may not be a compile-time
constant; furthermore, its value may depend on the file system,
see pathconf(3).

It's maybe to support strange OS like Hurd :-) (Hurd has no hardcoded limits).

Most of the time, the first realloc() should be enough.

[skrah]

For 2.x, unlimited path lengths were apparently introduced in bpo-2722.
This strategy does not work on Solaris and OpenBSD (bpo-9185).

FreeBSD also seems to support arbitrarily long paths. I would be somewhat
surprised though if anyone used them in practice. APUE (second edition)
uses PATH_MAX if it's available in limits.h.