Default value identity regression

[kayhayen]

BPO	27942
Nosy	@rhettinger, @larryhastings, @benjaminp, @ned-deily, @stevendaprano, @serhiy-storchaka, @1st1, @MojoVampire
Files	intern_string_constants.patch refleak.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-11-09.14:44:24.693>
created_at = <Date 2016-09-02.16:45:37.526>
labels = ['interpreter-core', '3.7', 'performance']
title = 'Default value identity regression'
updated_at = <Date 2017-12-19.08:50:39.586>
user = 'https://bugs.python.org/KayHayen'

bugs.python.org fields:

activity = <Date 2017-12-19.08:50:39.586>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-11-09.14:44:24.693>
closer = 'yselivanov'
components = ['Interpreter Core']
creation = <Date 2016-09-02.16:45:37.526>
creator = 'Kay.Hayen'
dependencies = []
files = ['44807', '45407']
hgrepos = []
issue_num = 27942
keywords = ['patch']
message_count = 15.0
messages = ['274257', '274262', '274263', '277313', '277323', '277359', '277748', '277749', '280395', '280399', '280400', '280401', '280402', '280403', '280404']
nosy_count = 11.0
nosy_names = ['rhettinger', 'larry', 'benjamin.peterson', 'ned.deily', 'steven.daprano', 'kayhayen', 'python-dev', 'serhiy.storchaka', 'yselivanov', 'Kay.Hayen', 'josh.r']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue27942'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[kayhayen]

Consider this:

def defaultKeepsIdentity(arg = "str_value"):
    print(arg is "str_value")

defaultKeepsIdentity()

This has been outputing "True" on every Python release I have seen so far, but not so on 3.6.0a4. Normally string values come from a "co_const" and will be "is" identical if used as literals in a module, but no longer in this case.

This seems wasteful at best, needlessly increasing the number of strings in usage.

Yours,
Kay

[stevendaprano]

Can confirm the expected behaviour (printing True) in Python 2.4 through 2.7, 3.3, Jython 2.5, and even venerable old Python 1.5 (where it prints 1).

But *not* IronPython 2.6, where it prints False.

In 3.6, the difference seems to be here:

py> f = defaultKeepsIdentity
py> f.__defaults__[0] is f.__code__.co_consts[1]
False
py> f.__defaults__[0] == f.__code__.co_consts[1]
True

This behaviour is not specified by the language. Caching and re-use of strings has always been subject to change. Nevertheless, perhaps it is time for this to be make a language feature: inside a function, any use of the same string literal should use the same object?

[serhiy-storchaka]

This likely is an interference of bpo-27095 with bpo-26148. I consider this as a regression. This causes not only a waste of memory, but can affect a performance, since comparing non-identical strings is slower than comparing identical strings.

[kayhayen]

Same with 3.6b1, still present.

[rhettinger]

It would be nice to get this fixed.

[serhiy-storchaka]

Proposed patch interns string constants recursively in tuples and frozensets. This fixes bpo-26148 and bpo-25981.

[python-dev]

New changeset 78bea78d9335 by Serhiy Storchaka in branch '3.5':
Issue bpo-27942: String constants now interned recursively in tuples and frozensets.
https://hg.python.org/cpython/rev/78bea78d9335

New changeset 0ce63a7651b9 by Serhiy Storchaka in branch '3.6':
Issue bpo-27942: String constants now interned recursively in tuples and frozensets.
https://hg.python.org/cpython/rev/0ce63a7651b9

New changeset 44af6bd21b94 by Serhiy Storchaka in branch 'default':
Issue bpo-27942: String constants now interned recursively in tuples and frozensets.
https://hg.python.org/cpython/rev/44af6bd21b94

New changeset 7cea3bf44acb by Serhiy Storchaka in branch '2.7':
Issue bpo-27942: String constants now interned recursively in tuples and frozensets.
https://hg.python.org/cpython/rev/7cea3bf44acb

[serhiy-storchaka]

Thanks Naoki and Josh for reviews.

Backported to 2.7 and 3.5 since the patch also fixes other issues.

Note that the result of arg is "str_value" is implementation detail. Other Python implementations may not intern string constants. Even CPython interns only strings constants consisting of ASCII alphanumerical characters.

[1st1]

The patch is causing refleaks:

test_ast leaked [98, 98, 98] references, sum=294
test_code leaked [2, 2, 2] references, sum=6

Please review the attached patch fixing that. Please review.