Code page decoder incorrectly handles input >2GiB

[serhiy-storchaka]

BPO	35372
Nosy	@vstinner, @serhiy-storchaka, @miss-islington
PRs	bpo-35372: Fix the code page decoder for input > 2 GiB. #10848 [3.7] bpo-35372: Fix the code page decoder for input > 2 GiB. (GH-10848) #10859 [3.6] bpo-35372: Fix the code page decoder for input > 2 GiB. (GH-10848) #10860

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2018-12-03.10:40:17.656>
created_at = <Date 2018-12-01.17:17:56.532>
labels = ['interpreter-core', '3.8', 'type-bug', '3.7']
title = 'Code page decoder incorrectly handles input >2GiB'
updated_at = <Date 2018-12-03.11:43:05.241>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2018-12-03.11:43:05.241>
actor = 'serhiy.storchaka'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2018-12-03.10:40:17.656>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2018-12-01.17:17:56.532>
creator = 'serhiy.storchaka'
dependencies = []
files = []
hgrepos = []
issue_num = 35372
keywords = ['patch']
message_count = 6.0
messages = ['330855', '330912', '330918', '330921', '330936', '330939']
nosy_count = 3.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'miss-islington']
pr_nums = ['10848', '10859', '10860']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue35372'
versions = ['Python 3.6', 'Python 3.7', 'Python 3.8']

[serhiy-storchaka]

>>> b = b'a'*(2**31-2)+b'\xff'*2
>>> x, y = codecs.code_page_decode(932, b, 'replace', True)
>>> len(x)
2
>>> x, y
('aa', 2147483648)

[serhiy-storchaka]

New changeset 4013c17 by Serhiy Storchaka in branch 'master':
bpo-35372: Fix the code page decoder for input > 2 GiB. (GH-10848)
4013c17

[miss-islington]

New changeset bdeb56c by Miss Islington (bot) in branch '3.7':
bpo-35372: Fix the code page decoder for input > 2 GiB. (GH-10848)
bdeb56c

[miss-islington]

New changeset 0f9b668 by Miss Islington (bot) in branch '3.6':
bpo-35372: Fix the code page decoder for input > 2 GiB. (GH-10848)
0f9b668

[vstinner]

Thanks for the fix ;-) I guess that nobody tried this code with a string longer than 2 GiB before you :-)

[serhiy-storchaka]

Decoding a 2 GiB string takes > 80 seconds on my computer and needs around 14 GiB of memory.