Problem with string concatenation and utf-8 cache.

[rpdKsa]

BPO	25709
Nosy	@malemburg, @birkenfeld, @terryjreedy, @pitrou, @vstinner, @larryhastings, @benjaminp, @ezio-melotti, @stevendaprano, @serhiy-storchaka, @eryksun
Files	greekbug.py: small script generating greek alphabet issue25709.patch issue25709_2.patch issue25709_3.patch issue25709_4.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/birkenfeld'
closed_at = <Date 2016-02-11.17:23:56.312>
created_at = <Date 2015-11-23.14:57:19.309>
labels = ['type-bug', 'library', 'release-blocker']
title = 'Problem with string concatenation and utf-8 cache.'
updated_at = <Date 2016-02-13.03:10:46.821>
user = 'https://bugs.python.org/rpdKsa'

bugs.python.org fields:

activity = <Date 2016-02-13.03:10:46.821>
actor = 'ned.deily'
assignee = 'georg.brandl'
closed = True
closed_date = <Date 2016-02-11.17:23:56.312>
closer = 'georg.brandl'
components = ['Library (Lib)']
creation = <Date 2015-11-23.14:57:19.309>
creator = '\xc3\x81rp\xc3\xa1d K\xc3\xb3sa'
dependencies = []
files = ['41139', '41140', '41141', '41142', '41146']
hgrepos = []
issue_num = 25709
keywords = ['patch']
message_count = 39.0
messages = ['255172', '255175', '255177', '255194', '255203', '255206', '255207', '255212', '255214', '255216', '255218', '255226', '255227', '255230', '255233', '255234', '255236', '255240', '255241', '255244', '255255', '255256', '255257', '255259', '255271', '255705', '255708', '255710', '255793', '255794', '255996', '256050', '260115', '260116', '260119', '260120', '260121', '260172', '260174']
nosy_count = 14.0
nosy_names = ['lemburg', 'georg.brandl', 'terry.reedy', 'pitrou', 'vstinner', 'larry', 'benjamin.peterson', 'ezio.melotti', 'steven.daprano', 'python-dev', 'serhiy.storchaka', 'eryksun', 'random832', '\xc3\x81rp\xc3\xa1d K\xc3\xb3sa']
pr_nums = []
priority = 'release blocker'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25709'
versions = ['Python 3.3']

[rpdKsa]

One of my students found this bug. For ascii characters it works as you expect, but for greek alphabet it works unexpectedly.
The program works correctly for Python 3.2.x but for 3.4.x and 3.5 it gives erroneous result.

[stevendaprano]

I'm afraid I'm unable to replicate this bug report in Python 3.4.

If you are able to replicate it, can you tell us the exact version number of Python you are using? Also, which operating system are you using?

[serhiy-storchaka]

Confirmed on IDLE.

>>> s = ''
>>> for i in range(5):
	s += '\xe0'
	print(s)

	
à
àà
àà
àà
àà
>>> s = ''
>>> for i in range(5):
	s += chr(0xe0)
	print(s)

	
à
àà
àà
àà
àà
>>> s = ''
>>> for i in range(5):
	s += '\u0107'
	print(s)

	
ć
ćć
ćć
ćć
ćć
>>> s = ''
>>> for i in range(5):
	s += chr(0x107)
	print(s)

ć
ć
ć
ć
ć

This issue can be related to details of IDLE's standard streams and RPC.

[serhiy-storchaka]

Here is reproducer without IDLE. Looks as pickle is a culprit.

>>> import pickle
>>> s = ''
>>> for i in range(5):
...         s += chr(0xe0)
...         print(len(s), s, s.encode(), repr(s))
...         print('   ', pickle.dumps(s))
... 
1 à b'\xc3\xa0' 'à'
    b'\x80\x03X\x02\x00\x00\x00\xc3\xa0q\x00.'
2 àà b'\xc3\xa0\xc3\xa0' 'àà'
    b'\x80\x03X\x04\x00\x00\x00\xc3\xa0\xc3\xa0q\x00.'
3 àà b'\xc3\xa0\xc3\xa0' 'ààà'
    b'\x80\x03X\x04\x00\x00\x00\xc3\xa0\xc3\xa0q\x00.'
4 àà b'\xc3\xa0\xc3\xa0' 'àààà'
    b'\x80\x03X\x04\x00\x00\x00\xc3\xa0\xc3\xa0q\x00.'
5 àà b'\xc3\xa0\xc3\xa0' 'ààààà'
    b'\x80\x03X\x04\x00\x00\x00\xc3\xa0\xc3\xa0q\x00.'

[random832]

I've looked at the raw bytes [through a ctypes pointer to id(s)] of a string affected by the issue, and decoded enough to be able to tell that the bad string has an incorrect UTF-8 length and data, which pickle presumably relies on.

HEAD............length..hash....flgs....wstr....u8len...u8ptr...wstrl...data....
010000003094ac6403000000ffffffffa40000000000000004000000d814480000000000e0e0e000
010000003094ac6403000000e5d0030ca400006500000000060000008814480000000000e0e0e000

I've omitted the UTF-8 data, but both were null-terminated after four and six bytes respectively.

[eryksun]

unicode_modifiable in Objects/unicodeobject.c should return 0 if there's cached PyUnicode_UTF8 data. In this case PyUnicode_Append won't operate in place but instead concatenate a new string.

[random832]

I can't reproduce without pickle. I did some further digging, though, and it *looks like*...

1. Pickle causes the built-in UTF-8 representation of a string to be populated, whereas encode('utf-8') does not. Can anyone think of any other operations that do this?
2. After the UTF-8 representation of the 2-character string is populated, concatenating a new character to it does not update or clear it.
3. However, it worked just fine with the 1-character string - concatenating it caused the UTF-8 representation to be cleared.

The actual operation that creates an inconsistent string is the concatenate operation, but it only happens with a string that has been "primed" by having its UTF-8 representation materialized.

[serhiy-storchaka]

Yes, I have came to the same as random832. String objects have "fast path" for concatenating, and in this path cached UTF8 representation is not cleaned. Pickle is one of simplest ways to reproduce this issue. May be it can be reproduced with compile() or type(), but these ways looks too heavyweighted to me.

Here is a patch that fixes strings concatenation.

[terryjreedy]

It would be good to get this in 3.4.4.