shutil.copy raises IsADirectoryError when the directory does not actually exist

[jerpint]

BPO	43219
Nosy	@gpshead, @ambv, @eryksun, @pablogsal, @miss-islington, @kulikjak, @akulakov, @jerpint
PRs	bpo-43219: shutil.copyfile, raise a less confusing exception instead of IsADirectoryError #27049 [3.10] bpo-43219: shutil.copyfile, raise a less confusing exception instead of IsADirectoryError (GH-27049) #27081 [3.9] bpo-43219: shutil.copyfile, raise a less confusing exception instead of IsADirectoryError (GH-27049) #27082 bpo-43219: handle Solaris as well #27257 [3.9] bpo-43219: skip Solaris in the test as well (GH-27257) #27267 [3.10] bpo-43219: skip Solaris in the test as well (GH-27257) #27268 [3.9] bpo-45234: Fix FileNotFound exception raised instead of IsADirectoryError in shutil.copyfile() (GH-28421) #28507 [3.10] bpo-45234: Fix FileNotFound exception raised instead of IsADirectoryError in shutil.copyfile() (GH-28421) #28508

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/gpshead'
closed_at = <Date 2021-07-10.15:03:04.309>
created_at = <Date 2021-02-14.05:16:28.392>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'shutil.copy raises IsADirectoryError when the directory does not actually exist'
updated_at = <Date 2021-10-04.19:18:42.219>
user = 'https://github.com/jerpint'

bugs.python.org fields:

activity = <Date 2021-10-04.19:18:42.219>
actor = 'pablogsal'
assignee = 'gregory.p.smith'
closed = True
closed_date = <Date 2021-07-10.15:03:04.309>
closer = 'gregory.p.smith'
components = ['Library (Lib)']
creation = <Date 2021-02-14.05:16:28.392>
creator = 'jerpint'
dependencies = []
files = []
hgrepos = []
issue_num = 43219
keywords = ['patch']
message_count = 20.0
messages = ['386932', '386933', '386935', '387134', '397060', '397061', '397119', '397235', '397236', '397237', '397865', '397893', '397897', '397898', '397901', '402022', '402373', '402386', '402394', '403155']
nosy_count = 9.0
nosy_names = ['gregory.p.smith', 'lukasz.langa', 'eryksun', 'pablogsal', 'miss-islington', 'kulikjak', 'Alex Grund', 'andrei.avk', 'jerpint']
pr_nums = ['27049', '27081', '27082', '27257', '27267', '27268', '28507', '28508']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43219'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[jerpint]

Issue: If you try to copy a file to a directory that doesn't exist using shutil.copy, a IsADirectory error is raised saying the directory exists.

This issue is actually caused when `open(not_a_dir, 'wb') is called on a non-existing dir.

Expected behaviour: Should instead raise NotADirectoryError
-----------------------------

Steps to reproduce:

[nav] In [1]: import os
...: from pathlib import Path
...: from shutil import copy
...:
...: tmp_file = '/tmp/some_file.txt'
...: Path(tmp_file).touch()
...: nonexistent_dir = 'not_a_dir/'
...: assert not os.path.exists(nonexistent_dir)
...: copy(tmp_file, nonexistent_dir)
---------------------------------------------------------------------------

IsADirectoryError                         Traceback (most recent call last)
<ipython-input-2-b0e0ec4f4875> in <module>
      7 nonexistent_dir = 'not_a_dir/'
      8 assert not os.path.exists(nonexistent_dir)
----> 9 copy(tmp_file, nonexistent_dir)

~/miniconda3/lib/python3.7/shutil.py in copy(src, dst, follow_symlinks)
243 if os.path.isdir(dst):
244 dst = os.path.join(dst, os.path.basename(src))
--> 245 copyfile(src, dst, follow_symlinks=follow_symlinks)
246 copymode(src, dst, follow_symlinks=follow_symlinks)
247 return dst

~/miniconda3/lib/python3.7/shutil.py in copyfile(src, dst, follow_symlinks)
119 else:
120 with open(src, 'rb') as fsrc:
--> 121 with open(dst, 'wb') as fdst:
122 copyfileobj(fsrc, fdst)
123 return dst

IsADirectoryError: [Errno 21] Is a directory: 'not_a_dir/'

[jerpint]

In fact, the issue seems to be coming from open() itself when opening a non-existent directory in write mode:

[nav] In [1]: import os
...: nonexixstent_dir = 'not_a_dir/'
...: assert not os.path.exists(nonexixstent_dir)
...: with open(nonexixstent_dir, 'wb') as fdst:
...: pass
---------------------------------------------------------------------------

IsADirectoryError                         Traceback (most recent call last)
<ipython-input-1-73d4010d6f34> in <module>
      2 dir_path = 'not_a_dir/'
      3 assert not os.path.exists(nonexixstent_dir)
----> 4 with open(nonexixstent_dir, 'wb') as fdst:
      5     pass

IsADirectoryError: [Errno 21] Is a directory: 'not_a_dir/'

[eryksun]

IsADirectoryError: [Errno 21] Is a directory: 'not_a_dir/'

The trailing slash forces the OS to handle "not_a_dir" as a directory [1].

A pathname that contains at least one non- <slash> character and that 
ends with one or more trailing <slash> characters shall not be resolved
successfully unless the last pathname component before the trailing 
<slash> characters names an existing directory or a directory entry 
that is to be created for a directory immediately after the pathname is
resolved. 

Mode "w" corresponds to low-level POSIX open() flags O_CREAT | O_TRUNC | O_WRONLY. If write access is requested for a directory, the open() system call must fail with EISDIR [2].

[EISDIR]
    The named file is a directory and oflag includes O_WRONLY or O_RDWR,
    or includes O_CREAT without O_DIRECTORY.

In most cases, opening a directory with O_CREAT also fails with E_ISDIR. POSIX does permit an implementation to create a directory with O_CREAT | O_DIRECTORY. In Linux, however, O_CREAT always creates a regular file, regardless of O_DIRECTORY, so open(pathname, O_CREAT | flags) always fails with EISDIR when pathname is an existing directory or names a directory by way of a trailing slash.

---
[1] https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#tag_04_13
[2] https://pubs.opengroup.org/onlinepubs/9699919799/functions/open.html

[eryksun]

I left this open in case someone wants to modify shutil.copy() and shutil.copy2() to raise a less misleading exception when dst names a non-existing directory such as 'not_a_dir/'. Failing with IsADirectoryError (errno EISDIR) is confusing since shutil.copy() and shutil.copy2() do support a destination directory.

Note that in Windows this case fails with EINVAL, which is at least less misleading than EISDIR. The EINVAL error is based on WinAPI ERROR_INVALID_NAME (123). CreateFileW() can create a directory if passed particular parameters, in which case a trailing slash is allowed. Otherwise it fails with ERROR_INVALID_NAME (123), unless it's an open-existing disposition, in which case it fails with ERROR_FILE_NOT_FOUND (2). This is specified in [MS-FSA] 2.1.5.1 [1]:

Phase 6 -- Location of file (final path component):

    Search ParentFile.DirectoryList for a Link where Link.Name or
    Link.ShortName matches FileNameToOpen. If such a link is found:

        Set File = Link.File.
        Set Open.File to File.
        Set Open.Link to Link.

    Else:

        If (CreateDisposition == FILE_OPEN || CreateDisposition ==
        FILE_OVERWRITE), the operation MUST be failed with
        STATUS_OBJECT_NAME_NOT_FOUND.

Phase 7 -- Type of stream to open:

    If PathName contains a trailing backslash:

        If StreamTypeToOpen is DataStream or
        CreateOptions.FILE_NON_DIRECTORY_FILE is TRUE, the operation
        MUST be failed with STATUS_OBJECT_NAME_INVALID.

NTAPI STATUS_OBJECT_NAME_NOT_FOUND and STATUS_OBJECT_NAME_INVALID map to WinAPI ERROR_FILE_NOT_FOUND and ERROR_INVALID_NAME.

---
[1] https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-fsa/8ada5fbe-db4e-49fd-aef6-20d54b748e40