obsolete macpath module dangerously broken and should be removed

[ned-deily]

BPO	9850
Nosy	@ronaldoussoren, @vstinner, @ned-deily, @merwok, @serhiy-storchaka, @yan12125, @DimitrisJim, @scottmacpherson, @Mariatta
PRs	bpo-9850: Deprecate the macpath module #1540 bpo-9850: Document macpath deprecation in Misc/NEWS #1590
Files	macpath_join_fix.patch: fix for macpath.join('', *comps) macpath_join_fix_with_test.patch: fix for macpath.join('', *comps) with a test 3.6-deprecate-macpath.patch 3.7-remove-macpath.patch 3.7-remove-macpath_ver2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2017-05-16.20:49:57.526>
created_at = <Date 2010-09-14.05:14:10.511>
labels = ['OS-mac', 'easy', '3.7']
title = 'obsolete macpath module dangerously broken and should be removed'
updated_at = <Date 2018-12-12.15:31:19.204>
user = 'https://github.com/ned-deily'

bugs.python.org fields:

activity = <Date 2018-12-12.15:31:19.204>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2017-05-16.20:49:57.526>
closer = 'vstinner'
components = ['macOS']
creation = <Date 2010-09-14.05:14:10.511>
creator = 'ned.deily'
dependencies = []
files = ['22929', '22943', '44930', '44931', '44932']
hgrepos = []
issue_num = 9850
keywords = ['patch', 'easy']
message_count = 33.0
messages = ['116367', '116387', '116388', '116389', '116517', '116973', '142310', '142365', '142397', '217238', '217901', '227687', '227689', '247128', '247242', '277844', '277892', '277893', '278102', '293443', '293458', '293692', '293693', '293694', '293695', '293696', '293697', '293698', '293700', '293704', '293706', '293785', '331701']
nosy_count = 12.0
nosy_names = ['ronaldoussoren', 'vstinner', 'ned.deily', 'eric.araujo', 'jesstess', 'chortos', 'python-dev', 'serhiy.storchaka', 'yan12125', 'Jim Fasarakis-Hilliard', 'macpherson.scott', 'Mariatta']
pr_nums = ['1540', '1590']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue9850'
versions = ['Python 3.7']

[ned-deily]

The macpath module in the standard library purports to supply "the Mac OS 9 (and earlier) implementation of the os.path module. It can be used to manipulate old-style Macintosh pathnames on Mac OS X (or any other platform). The following functions are available in this module: normcase(), normpath(), isabs(), join(), split(), isdir(), isfile(), walk(), exists(). For other functions available in os.path dummy counterparts are available."

Old-style Mac pathnames are not further documented - in fact, the above quote is the entire external documentation for the module - but they are ones using colon separators, like ("MacHD:Users:nad:macpath_test:file"). These style path names were initially supported on Mac OS X by compatibility APIs for programs ported from classic Mac OS but those interfaces have long been deprecated in OS X and in most cases are not available in 64-bit execution mode.

Even if one did have a need to use the obsolete old-style paths, the macpath module is currently practically unusable for anything other than simple character manipulations of the path. Nearly all of the functions that actually call OS routines are broken in one or more ways.

Those that do make file system calls are calling APIs that are expecting normal posix-style ('/' separated paths) incorrectly with old-style (":) paths (ispath, isdir, lexists, etc) which means they only give correct results in the trivial case of unqualified file or directory names, i.e. those in the working directory.

The islink() function confusingly is testing whether a path points to a Finder Alias file (not a symlink). Unfortunately, the Carbon API used for that test does not exist in a 64-bit version and all of the Python Carbon modules were removed in Python 3.

$ arch -i386 python2.7 -c 'import macpath; print(macpath.islink("alias_to_file"))'
1
$ arch -x86_64 python2.7 -c 'import macpath; print(macpath.islink("alias_to_file"))'
False
$ python3.2 -c 'import macpath; print(macpath.islink("alias_to_file"))'
False

The underlying import errors are being masked by "try"'s:

$ arch -i386 python2.7 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
$ arch -x86_64 python2.7 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: 'module' object has no attribute 'ResolveAliasFile'
$ python3.2 -c 'import Carbon.File; Carbon.File.ResolveAliasFile'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named Carbon.File

The realpath function is also broken in 2.7, calling Carbon.File.FSResolveAliasFile with a ":" separated path when it expects a "/" path, and is totally broken in 3.x (no Carbon modules).

While macpath *could* be repaired by proper documentation, fixing the mishmash of path types internally, and supplying C wrappers and/or alternative 64-bit APIs, it does not seem worth the effort as these style paths are seldom encountered in modern code. Considering how broken the module currently is and given that it probably was simply overlooked in the Python 2 to 3 transition, I think a case could be made for immediate removal prior to Python 3.2 release and even for a 3.1.x maintenance release. Short of that, it should be cleared documented as deprecated in 3.2, 3.1, and 2.7 along with warnings about broken functionality along with added DeprecationWarnings to the module.

I can write up patches to do that depending on what the consensus is.

[ronaldoussoren]

I'm -0 on removing the module wholesale because parts of OSX still use MacOS9 style paths (in particular some Carbon APIs and AppleScript).

I'm +1 on removing all functions that aren't path manipulation code, that is 'macpath.join' would stay while 'macpath.islink' would go, and fixing the documentation.

And to be complete: I'm -1 on fixing macpath.islink and other functions that access a life filesystem because the macpath module is only meant to be used for that on MacOS9 and that platform is no longer supported. Path manipulation code is still useful.

BTW. The module cannot be removed from 2.7 and 3.1 due to API stability requirements.

[vstinner]

The solution may be different depending on Python version. I propose to keep macpath in Python 2.7, just because it's too late to change such thing in Python2. But we may mark macpath as deprecated, eg. "macpath will be removed in Python 3.2" (see above). I suppose that the same treatement can be used on Python 3.1 (maybe with a different warning type/message).

But for Python 3.2, I suggest to drop macpath. Said differently: drop Mac OS 9 support. If you really need Mac OS 9 support, continue to use Python 2.7. If I understood correctly, macpath is not used for os.path in Python 3.2, and it is broken (at least on 64 bits build).

But well, I am not a Mac programmer, so I don't know if Mac OS 9 is still commonly installed or not.

[ronaldoussoren]

MacOS9 is already unsupported, except for macpath all traces of OS9 support are gone with the possible exception of distutils (I removed traces of OS9 support code in 3.2 and those got restored when Tarek replaced distutils by the version from the 3.1 branch, I don't remember if I redid my OS9-removal work).

As I wrote earlier the path manipulation code (macpath.join, macpath.split etc.) are still useful because OS9-style paths are still used in some parts of OSX.

[ned-deily]

Patches in progress.

[merwok]

I agree with Ronald’s proposal.

(Re “I don't remember if I redid my OS9-removal work”: Looks like you didn’t.)

[chortos]

I fully agree with Ronald’s proposal. And for a start, here is a trivial patch that fixes macpath.join('', ...) [at the moment it just returns its last argument]. By the way, this fix is probably eligible for inclusion in Python 2.7 too.

I also have ‘implement macpath.relpath’ in my to-do list, although in practice this means nothing.

[merwok]

Thanks for the patch. Is the function already tested or does someone need to add a test?

[chortos]

It is already tested but here is a new version of the patch that expands the existing test to cover the situation that was broken (plus another one).

[jesstess]

Thanks for writing up this issue, ned.deily, and thanks for providing a patch, chortos.

I couldn't find documentation clearly specifying what the correct behavior of macpath.join should be (i.e. what are the exact rules for leading and trailing colons), but comparing the old and updated behavior against the new tests, this patch does make the function's behavior more consistent.

• The patch passes make patchcheck.
• The full test suite passes with this patch.

=> commit review

[ned-deily]

The patch appears fine but it really doesn't have anything to do with the gist of this issue. The problem remains that much of macpath is fundamentally broken. In the intervening years since the issue was opened, I contend that any need for OS 9 style paths (aka HFS paths) has diminished even further so I would still support starting in Python 3.5 the deprecation and removal process for this module. But, as long as it remains, to address the current problems at a minimum a patch would be needed to remove or raise exceptions for all of the macpath functions that make file system calls and to update the documentation.

[python-dev]

New changeset 2ae2ca9d2b66 by Serhiy Storchaka in branch '2.7':
Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
https://hg.python.org/cpython/rev/2ae2ca9d2b66

New changeset 54987723de99 by Serhiy Storchaka in branch '3.4':
Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
https://hg.python.org/cpython/rev/54987723de99

New changeset e29866cb6b98 by Serhiy Storchaka in branch 'default':
Issue bpo-9850: Fixed macpath.join() for empty first component. Patch by
https://hg.python.org/cpython/rev/e29866cb6b98