The _csv module can't be garbage-collected after _csv.register_dialect is called

[encukou]

BPO	44116
Nosy	@encukou, @corona10, @pablogsal, @miss-islington, @erlend-aasland
PRs	bpo-44116: Work around module unloading refleak to unblock buildbots #26068 bpo-44116: Add GC support to _csv heap types #26074 [3.10] bpo-44116: Add GC support to _csv heap types (GH-26074) #26081
Files	gc.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-12.19:32:53.480>
created_at = <Date 2021-05-12.13:34:11.112>
labels = ['extension-modules', '3.11']
title = "The _csv module can't be garbage-collected after _csv.register_dialect is called"
updated_at = <Date 2021-05-12.20:31:24.940>
user = 'https://github.com/encukou'

bugs.python.org fields:

activity = <Date 2021-05-12.20:31:24.940>
actor = 'corona10'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-12.19:32:53.480>
closer = 'pablogsal'
components = ['Extension Modules']
creation = <Date 2021-05-12.13:34:11.112>
creator = 'petr.viktorin'
dependencies = []
files = ['50038']
hgrepos = []
issue_num = 44116
keywords = ['patch']
message_count = 16.0
messages = ['393508', '393512', '393518', '393520', '393521', '393522', '393523', '393525', '393526', '393527', '393529', '393530', '393531', '393543', '393545', '393546']
nosy_count = 5.0
nosy_names = ['petr.viktorin', 'corona10', 'pablogsal', 'miss-islington', 'erlendaasland']
pr_nums = ['26068', '26074', '26081']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue44116'
versions = ['Python 3.11']

[encukou]

After _csv.register_dialect is called, the csv module is alive even after it's removed from sys.modules. It should be garbage-collected.
(It's not that big a deal: unloading _csv isn't something users should do. But it might be hiding a deeper issue.)

The following reproducer (for a debug build of Python) shows an increasing number of refcounts. (Importing csv is the easiest way to call _csv._register_dialect with a proper argument):

import sys
import gc

for i in range(10):
    import csv
    del sys.modules['_csv']
    del sys.modules['csv']
    del csv
    gc.collect()

    print(sys.gettotalrefcount())

[encukou]

Hm, a similar thing apparently happens with urllib.request.

[erlend-aasland]

Could it be that the _csv heap types are not garbage collected? Ref. bpo-42972.

[erlend-aasland]

Adding GC to _csv types:
$ cat
import sys
import gc

for i in range(10):
    import csv
    del sys.modules['_csv']
    del sys.modules['csv']
    del csv
    gc.collect()

    print(sys.gettotalrefcount())
$ ./python.exe bug.py
73164
73164
73166
73166
73166
73166
73166
73166
73166
73166

[erlend-aasland]

I created a quick-and-dirty patch. I can clean it up and make it into a PR if you want.

[encukou]

*facepalm* Yes, that's it.

If you have the time now, could you send he PR?

[erlend-aasland]

Sure, I’ll do it after dinner :)

[encukou]

The urllib.request one was caused by _hashlib, see #70260.

[erlend-aasland]

I don't understand this. After applying PR-26074, test_csv now leaks memory/refs:

$ ./python.exe -m test -R : test_csv
0:00:00 load avg: 1.18 Run tests sequentially
0:00:00 load avg: 1.18 [1/1] test_csv
beginning 9 repetitions
123456789
.........
test_csv leaked [3928, 3924, 3924, 3924] references, sum=15700
test_csv leaked [1666, 1664, 1664, 1664] memory blocks, sum=6658
test_csv failed

== Tests result: FAILURE ==

1 test failed:
test_csv

Total duration: 4.9 sec
Tests result: FAILURE

Any ideas?

[erlend-aasland]

Also, for some reason two first iterations of the reproducer prints 2 less ref counts.

[encukou]

Changes to _csv.Error should not be necessary, there everything is handled by the superclass.