os.replace is not cross-platform: at least improve documentation

[hancos]

BPO	46003
Nosy	@ericvsmith, @eryksun

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-12-08.09:08:26.099>
created_at = <Date 2021-12-06.21:20:05.764>
labels = ['invalid', 'type-bug', '3.8']
title = 'os.replace is not cross-platform: at least improve documentation'
updated_at = <Date 2021-12-08.09:46:58.400>
user = 'https://bugs.python.org/hancos'

bugs.python.org fields:

activity = <Date 2021-12-08.09:46:58.400>
actor = 'eric.smith'
assignee = 'none'
closed = True
closed_date = <Date 2021-12-08.09:08:26.099>
closer = 'hancos'
components = []
creation = <Date 2021-12-06.21:20:05.764>
creator = 'hancos'
dependencies = []
files = []
hgrepos = []
issue_num = 46003
keywords = []
message_count = 6.0
messages = ['407864', '407876', '407884', '407961', '407974', '408003']
nosy_count = 3.0
nosy_names = ['eric.smith', 'eryksun', 'hancos']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue46003'
versions = ['Python 3.8']

[hancos]

When using threads, os.replace gives no errors on Linux. On Windows however you get PermissionErrors like: '[WinError 5] Access is denied' for os.replace and '[Errno 13] Permission denied' when reading the os.replace file.
The only way I could get this to work was to add retries with delay.
At least the documentation should be improved.

[ericvsmith]

I suspect anti-virus software.

[eryksun]

The os module tries to avoid documenting low-level OS behaviors. It would be unreliable and difficult to maintain.

In the case of os.replace(), in Windows it calls MoveFileExW() with the flag MOVEFILE_REPLACE_EXISTING [1]. The source and destination paths must be on the same volume. The caller must have permission to delete the source path and, if it already exists, the destination path. The caller must have permission to add a file or directory to the destination directory. Existing opens of the source path must share delete access. If the source path is a directory, none of the files and directories in its tree is allowed to be open. If the destination path already exists, it cannot be a directory, a readonly file, or mapped as a process image (i.e. a data mapping is allowed, but an executing EXE or DLL is disallowed). Currently, existing opens of the destination path are not allowed, even if they share delete access.

In Windows 10+, there's new support at the NT system call level (i.e. the layer beneath the Windows API) to support replacing a file that has existing opens, if the file system supports it (e.g. NTFS, ReFS). Delete access is still required, so the opens must share delete access. MoveFileExW() has not been updated yet to use this new capability. If and when it's supported, the requirement for existing opens to share delete access means it won't help in general. Most Windows programs, including Python, do not share delete access on open files. To share delete access in Python, one can use a custom opener function that calls CreateFileW() and wraps the OS handle in an fd via msvcrt.open_osfhandle().

---
[1] https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-movefileexw

[hancos]

Thank you for your replies. I have been reading more and conclude that I/O programming on Windows and Linux are two different things.

Python is not cross-platform in the sense that a Python program always works on different operating systems but that you can make it work on different operating systems (most of the time).

My application uses the PyPi package cachelib. The file system interface is implemented in this file:

https://github.com/pallets/cachelib/blob/main/src/cachelib/file.py

This package uses os.replace and other file I/O methods.

From what I understand is that this package must be modified to also handle WinError 5 and Errno 13 if you want to use it on Windows,
meaning more specific exception handling and adding retries and a delay for every I/O related function accessing the os.replace file.

Please correct me if I am wrong.

[ericvsmith]

As they say, there's no such thing as "portable software", only "software that has been ported". Especially in an area like file I/O: once you move beyond simple "one process opens, writes, and closes; and another process then opens, reads, and closes", there are a lot of platform-specific issues. Python does not try to abstract away all possible file I/O issues.

[hancos]

Thank you again, closing this issue.