xml.ElementTree iterparse filehandle left open

[vkisforever]

BPO	43292
Nosy	@vadmium, @serhiy-storchaka, @tonybaloney, @miss-islington, @jacobtylerwalls
PRs	bpo-43292: Fix file leak in ET.iterparse() when not exhausted #31696 [3.9] bpo-43292: Fix file leak in ET.iterparse() when not exhausted (GH-31696) #31720 [3.10] bpo-43292: Fix file leak in ET.iterparse() when not exhausted (GH-31696) #31721
Files	ETiterparse.py: iterparse issue recreator

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-03-07.11:50:57.033>
created_at = <Date 2021-02-22.04:11:29.388>
labels = ['3.9', '3.10', 'performance', 'expert-XML', '3.11', 'library']
title = 'xml.ElementTree iterparse filehandle left open'
updated_at = <Date 2022-03-07.11:50:57.032>
user = 'https://bugs.python.org/vkisforever'

bugs.python.org fields:

activity = <Date 2022-03-07.11:50:57.032>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2022-03-07.11:50:57.033>
closer = 'serhiy.storchaka'
components = ['Library (Lib)', 'XML']
creation = <Date 2021-02-22.04:11:29.388>
creator = 'vkisforever'
dependencies = []
files = ['49826']
hgrepos = []
issue_num = 43292
keywords = ['patch']
message_count = 10.0
messages = ['387494', '387495', '387496', '387497', '387498', '387571', '388457', '414647', '414653', '414658']
nosy_count = 7.0
nosy_names = ['martin.panter', 'serhiy.storchaka', 'anthonypjshaw', 'miss-islington', 'vkisforever', 'dtom9424', 'jacobtylerwalls']
pr_nums = ['31696', '31720', '31721']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue43292'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[vkisforever]

Standard library xml.ElementTree - iterparse does not close filehandle properly when breaking out of a for loop iterating with iterparse. The issue only occurs when the user passes a "string" as a source object instead of a filehandle (that is because a user's filehandle can safely be closed in a context manager as shown in the attachment).

[tonybaloney]

which version of Python were you doing this on?

The function in question is https://github.com/python/cpython/blob/master/Lib/xml/etree/ElementTree.py#L1233-L1278

[vkisforever]

ran python 3.9.2 on windows 10

1. What did you do?
   Tried to remove folder that contains a xml file that was worked on by ElementTree iterparse with argument source='filename' (string type input)

2. What happened?
   python was not able to remove the folder due to file lock still held by iterparse.
3. What did you expect to happen
   iterparse was supposed to close the filehandle such that the user could then write/move/delete the xml file worked on, however python gave a PermissionError:

Traceback (most recent call last):
  File "C:\Users\vkisf\Desktop\00_MYSTUFF\01_SOFTWARE\01_PYTHON\01_GITHUB\pycascades2021\ETiterparse.py", line 43, in <module>
    shutil.rmtree('subdir2')
  File "C:\Users\vkisf\Desktop\00_MYSTUFF\01_SOFTWARE\01_PYTHON\01_GITHUB\pycascades2021\cpython\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\vkisf\Desktop\00_MYSTUFF\01_SOFTWARE\01_PYTHON\01_GITHUB\pycascades2021\cpython\lib\shutil.py", line 618, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\vkisf\Desktop\00_MYSTUFF\01_SOFTWARE\01_PYTHON\01_GITHUB\pycascades2021\cpython\lib\shutil.py", line 616, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'subdir2\\app.xml'

[tonybaloney]

Example script attached works perfectly on macOS

python3.9 test.py
[('default', 'http://schemas.openxmlformats.org/officeDocument/2006/extended-properties'), ('vt', 'http://schemas.openxmlformats.org/officeDocument/2006/docPropsVTypes')]
[('default', 'http://schemas.openxmlformats.org/officeDocument/2006/extended-properties'), ('vt', 'http://schemas.openxmlformats.org/officeDocument/2006/docPropsVTypes')]

[vkisforever]

Confirmed that no apps were using the created xml file or folder. Windows Explorer was closed for all folders only the terminal shell was open. Issue still persists on windows 10 OS.

[dtom9424]

"erase all files"