bpo-1635741: Fix refleaks of encodings module by removing the encodings._aliases

[shihai1991]

Fix refleaks of encodings._aliases by using encodings.aliases directly in encodings.search_function.

Co-authored-by: Victor Stinner [email protected]

https://bugs.python.org/issue1635741

[shihai1991]

I use the test case of https://bugs.python.org/issue1635741#msg355187 to test the refleaks in debug mode.

Before this PR:
sys.gettotalrefcount: 14288
sys.gettotalrefcount: 18042
sys.gettotalrefcount: 21796
sys.gettotalrefcount: 25550
sys.gettotalrefcount: 29304
sys.gettotalrefcount: 33058
sys.gettotalrefcount: 36812
sys.gettotalrefcount: 40566
sys.gettotalrefcount: 44320
sys.gettotalrefcount: 48074

After this PR:
sys.gettotalrefcount: 13641
sys.gettotalrefcount: 16748
sys.gettotalrefcount: 19855
sys.gettotalrefcount: 22962
sys.gettotalrefcount: 26069
sys.gettotalrefcount: 29176
sys.gettotalrefcount: 32283
sys.gettotalrefcount: 35390
sys.gettotalrefcount: 38497
sys.gettotalrefcount: 41604

[shihai1991]

@vstinner Hi, victor. Pls take a look if you have free time, thanks.

[ghost]

Don't understand the problem.
This statement should be placed at below so that it does not affect the performance of the cache.

[shihai1991]

Don't understand the problem.

Thanks for your comment. It will affect the encodings module's refcount in C level and reduce the refleaks.

This statement should be placed at below so that it does not affect the performance of the cache.

MAYBE removing this line and using aliases.aliases to replace _aliases is fine too :)

[shihai1991]

Looking this comment: #21896 (comment)

[ghost]

Looking this comment: #21896 (comment)

The usage of aliases.aliasesis is very normal, maybe the root of the problem is not here.

[vstinner]

I don't see how using encodings._aliases in search_function() creates a "reference leak". A leak is when calling a function multiple times leaks memory. Here, there is no leak.

Maybe you're talking about a "reference cycle".

I guess that you're trying to clear variables at exit.

You should try to trigger an explicit GC collection after calling PyInterpreterState_Clear(). In finalize_interp_clear(), try to replace:

    /* Trigger a GC collection on subinterpreters*/
    if (!is_main_interp) {
        _PyGC_CollectNoFail();
    }

with:

    // Last explicit GC collection
    _PyGC_CollectNoFail();

(without this change)

Does it fix your issue?

PyInterpreterState_Clear() clears the reference to the search function: Py_CLEAR(interp->codec_search_path).

[shihai1991]

Maybe you're talking about a "reference cycle".

Thanks, victor. "reference cycle" would be more exact. And I will try your idea in my interpreter.

[shihai1991]

    /* Trigger a GC collection on subinterpreters*/
    if (!is_main_interp) {
        _PyGC_CollectNoFail();
    }

with:

    // Last explicit GC collection
    _PyGC_CollectNoFail();

Oh, amazing result:

sys.gettotalrefcount: 10537
sys.gettotalrefcount: 10540
sys.gettotalrefcount: 10543
sys.gettotalrefcount: 10546
sys.gettotalrefcount: 10549
sys.gettotalrefcount: 10552
sys.gettotalrefcount: 10555
sys.gettotalrefcount: 10558
sys.gettotalrefcount: 10561
sys.gettotalrefcount: 10564

the pr in: #21902

[shihai1991]

Pablo created this PR(don't calling explict collection in main interpreter): #17457
I am not sure I missed some info or not~

[vstinner]

Since #17457 is merged, is this PR still relevant/useless? If not, please close it.

[shihai1991]

Since #17457 is merged, is this PR still relevant/useless? If not, please close it.

#17457 is worked, thanks.