os.path.realpath() produces unexpected results when run on a path created using the /proc/self/fd/ mechanism

[sirosen]

Bug report

(editor's note: the description below indicates that realpath() is unaffected, but that's not correct - see comment)

If a file descriptor is created in /proc/self/fd/ and then passed to pathlib.Path, the resolve() method produces a result which is no longer a real and valid path to the file.
MRE, including os.path.realpath for comparison:

import os, pathlib
fd = os.memfd_create("myfd")
print(pathlib.Path(f"/proc/self/fd/{fd}").resolve())
# /memfd:myfd (deleted)
print(os.path.realpath("/proc/self/fd/4"))
# /proc/227671/fd/4

This can occur easily if another program uses the /proc/self/fd/ mechanism to pass temporary files to a python application.
For example, using ZSH on linux:

$ python -c 'import pathlib, sys; print(pathlib.Path(sys.argv[1]).resolve())' <(echo 'hi')
/proc/234888/fd/pipe:[331484707]

I looked at pathlib.py and in 3.11 it looks like it's primarily relying on realpath, so I'm not clear on where or how the discrepancy gets introduced. I'm sure I don't understand something about pathlib which explains.

Your environment

• CPython versions tested on: 3.10, 3.11
• Operating system and architecture: Linux

[eryksun]

The proc filesystem on Linux supports symlinks to anonymous files. These symlinks can be traversed even though the target path that's returned by readlink() doesn't exist, such as "pipe:[331484707]" or, for a deleted file, "/path/to/filename (deleted)".

You can try a workaround based on passing strict=True. If resolving strictly fails but stat() succeeds, resolve the parent path strictly and join the unresolved symlink to the result. Of course, it isn't strictly a 'real' path, but that's not possible in this case.

[sirosen]

Thanks for answering, and for suggesting workarounds for this scenario!
For now, I've put in place a regex match for /proc/(self|\d+)/fd/\d+ since I'm not aware of other cases, but I'll also give thought to the strict=True usage. (My regex match doesn't, for example, handle a symlink to one of these files.)

This makes me see this not as a bug in pathlib, but it's simultaneously not what I expected as a user.
The basic form of usage is wanting to both resolve the path and also read a file. Something like

path = pathlib.Path(filename).resolve()
print(f"contents of {path}:")
with open(path) as fp:
    print(fp.read())

That works in most scenarios.
It's pretty surprising that open(path) could fail in that example where open(filename) would have worked.

Is there any possible enhancement here to pathlib? I would expect most users of resolve() are treating it as a way of getting a fully normalized path to a file which can be opened and used. Any other usage is likely more niche. At the same time, I can't think of a backwards compatible way of introducing my desired behavior other than to add a new method or option.

[barneygale]

I think there's a typo in the repro code - the fd value should be included in the realpath() call:

import os, pathlib
fd = os.memfd_create("myfd")
print(pathlib.Path(f"/proc/self/fd/{fd}").resolve())
# /memfd:myfd (deleted)
print(os.path.realpath(f"/proc/self/fd/{fd}"))
# /memfd:myfd (deleted)

With the corrected code I've found resolve() and realpath() to be consistent from 3.8 through 3.13.

If there's a bug here, it's in realpath() and not pathlib, so I'm removing the topic-pathlib label and re-titling the bug.

[sirosen]

Yep, that's a good catch! The MRE is wrong. I can't say offhand how that got mixed up on my end but please pay it no mind.

Even having learned more from this thread, I still think this is a bit of a "gotcha" about realpath(). But it's not clear what kind of enhancement (behavior-wise or docs) would help.

In the simplest, most naive terms, what I'd like is a variant of realpath() which follows symlinks, but will stop short of resolving a usable path to an unusable one (like these /proc/ items); perhaps defined by stat working on the path. os.path.realpath2() or os.path.realpath(..., do_the_thing_from_issue_99390=True) seem like unacceptable solutions for that even if it were easy to implement.

I'm coming around to the thought -- given that I was mistaken about os.path.realpath() being different from pathlib.Path.resolve() -- that this should be closed with no action. I'll refrain from closing right away, but I don't want to take up people's time with a non-actionable enhancement request.