Multiple tests are leaking references in AMD64 Windows8.1 Refleaks 3.x and x86 Gentoo Refleaks 3.x buildbots

[pablogsal]

BPO	37954
Nosy	@ncoghlan, @ambv, @zooba, @pablogsal, @miss-islington
PRs	bpo-37954: Fix reference leak in the symtable #15514 [3.8] bpo-37954: Fix reference leak in the symtable (GH-15514) #15518

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-08-27.19:35:24.123>
created_at = <Date 2019-08-26.14:30:13.112>
labels = ['3.8', '3.9', 'release-blocker']
title = 'Multiple tests are leaking references in AMD64 Windows8.1 Refleaks 3.x and x86 Gentoo Refleaks 3.x buildbots'
updated_at = <Date 2019-08-29.11:57:37.967>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2019-08-29.11:57:37.967>
actor = 'ncoghlan'
assignee = 'none'
closed = True
closed_date = <Date 2019-08-27.19:35:24.123>
closer = 'pablogsal'
components = []
creation = <Date 2019-08-26.14:30:13.112>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 37954
keywords = ['patch']
message_count = 10.0
messages = ['350533', '350534', '350536', '350538', '350539', '350542', '350543', '350547', '350644', '350765']
nosy_count = 5.0
nosy_names = ['ncoghlan', 'lukasz.langa', 'steve.dower', 'pablogsal', 'miss-islington']
pr_nums = ['15514', '15518']
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue37954'
versions = ['Python 3.8', 'Python 3.9']

[pablogsal]

BUILDBOT FAILURE REPORT
=======================

Builder name: AMD64 Windows8.1 Refleaks 3.x
Builder url: https://buildbot.python.org/all/#/builders/80/
Build url: https://buildbot.python.org/all/#/builders/80/builds/683

Failed tests
------------

Test leaking resources
----------------------

• test_tools is leaking references

• test_exceptions is leaking references

• test_compile is leaking references

• test_trace is leaking references

• test_dis is leaking references

• test_builtin is leaking references

• test_syntax is leaking references

• test_named_expressions is leaking memory blocks

• test_syntax is leaking memory blocks

• test_symtable is leaking references

• test_typing is leaking references

• test_future is leaking references

• test_global is leaking references

• test_opcodes is leaking references

• test_grammar is leaking references

• test_scope is leaking references

• test_named_expressions is leaking references

• test_ast is leaking references

• test_ast is leaking memory blocks

Build summary
-------------

== Tests result: FAILURE then FAILURE ==

373 tests OK.

10 slowest tests:

• test_multiprocessing_spawn: 25 min 17 sec
• test_asyncio: 19 min 45 sec
• test_mailbox: 15 min 38 sec
• test_distutils: 14 min 8 sec
• test_concurrent_futures: 13 min 58 sec
• test_zipfile: 9 min 36 sec
• test_venv: 8 min 8 sec
• test_compileall: 6 min 37 sec
• test_lib2to3: 6 min 11 sec
• test_regrtest: 5 min 43 sec

16 tests failed:
test_ast test_builtin test_compile test_dis test_exceptions
test_future test_global test_grammar test_named_expressions
test_opcodes test_scope test_symtable test_syntax test_tools
test_trace test_typing

30 tests skipped:
test_curses test_dbm_gnu test_dbm_ndbm test_devpoll test_epoll
test_fcntl test_fork1 test_gdb test_grp test_ioctl test_kqueue
test_multiprocessing_fork test_multiprocessing_forkserver test_nis
test_openpty test_ossaudiodev test_pipes test_poll test_posix
test_pty test_pwd test_readline test_resource test_spwd
test_syslog test_threadsignals test_wait3 test_wait4
test_xxtestfuzz test_zipfile64

20 re-run tests:
test_ast test_asyncgen test_builtin test_compile test_compileall
test_dis test_exceptions test_future test_global test_grammar
test_named_expressions test_opcodes test_scope test_symtable
test_syntax test_tarfile test_threading test_tools test_trace
test_typing

Total duration: 1 hour 21 min

Tracebacks
----------

Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af9f0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af3c0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652afe10> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652afb50> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af730> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af890> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af7e0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652afec0> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb61c51f70> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request


Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\proactor_events.py", line 765, in _loop_self_reading
    f.result()  # may raise
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 808, in _poll
    value = callback(transferred, key, ov)
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 457, in finish_recv
    raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
.Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af260> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Cancelling an overlapped future failed
future: <_OverlappedFuture pending overlapped=<pending, 0xbb652af310> cb=[BaseProactorEventLoop._loop_self_reading()]>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 66, in _cancel_overlapped
    self._ov.cancel()
OSError: [WinError 6] The handle is invalid
Error on reading from the event loop self pipe
loop: <ProactorEventLoop running=True closed=False debug=False>
Traceback (most recent call last):
  File "D:\buildarea\3.x.ware-win81-release.refleak\build\lib\asyncio\windows_events.py", line 453, in finish_recv
    return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request

Current builder status
----------------------

The builder is failing currently

Commits
-------

Other builds with similar failures
----------------------------------

• https://buildbot.python.org/all/#/builders/1/builds/694
• https://buildbot.python.org/all/#/builders/224/builds/70
• https://buildbot.python.org/all/#/builders/223/builds/84

[pablogsal]

Bisecting points at commit 5dbe0f5 as the commit that introduced the reference leaks

[pablogsal]

See also https://bugs.python.org/issue37947

[pablogsal]

New changeset 4901dc4 by Pablo Galindo in branch 'master':
bpo-37954: Fix reference leak in the symtable (GH-15514)
4901dc4

[pablogsal]

Scheduled two custom builds of the 3.x refleak buildbots:

https://buildbot.python.org/all/#/builders/80/builds/684
https://buildbot.python.org/all/#/builders/1/builds/695

[miss-islington]

New changeset ed8af33 by Miss Islington (bot) in branch '3.8':
bpo-37954: Fix reference leak in the symtable (GH-15514)
ed8af33

[pablogsal]

Scheduled another two custom builds on the 3.8 branch:

https://buildbot.python.org/all/#/builders/223/builds/85
https://buildbot.python.org/all/#/builders/224/builds/71

[zooba]

Great work, Pablo! This does *not* look like it was an easy one to track down.

[pablogsal]

Thank you very much, Steve! This was indeed a bit tricky :)

The last refleak buildbot is green again, so closing this.

[ncoghlan]

Thank you Pablo!

Even having seen your fix, it still took me a couple of rescans of the original PR to figure out exactly how I'd broken it in the first place (refactoring to call an existing function without noticing that the replaced code included an extra DECREF).