can't interpolate byte string with \x00 before replacement identifier

[NickHuber]

BPO	29714
Nosy	@vstinner, @serhiy-storchaka, @zhangyangyu
PRs	bpo-29714: Fix a regression that bytes format may fail when containing zero bytes inside. #499 [3.6] bpo-29714: Fix a regression that bytes format may fail when containing zero bytes inside. #504 [Do Not Merge] Sample of CPython life with blurb. #703 [Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zhangyangyu'
closed_at = <Date 2017-03-06.10:18:22.945>
created_at = <Date 2017-03-03.19:05:28.034>
labels = ['interpreter-core', 'type-bug', '3.7']
title = "can't interpolate byte string with \\x00 before replacement identifier"
updated_at = <Date 2017-03-31.16:36:26.220>
user = 'https://bugs.python.org/NickHuber'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:26.220>
actor = 'dstufft'
assignee = 'xiang.zhang'
closed = True
closed_date = <Date 2017-03-06.10:18:22.945>
closer = 'xiang.zhang'
components = ['Interpreter Core']
creation = <Date 2017-03-03.19:05:28.034>
creator = 'Nick Huber'
dependencies = []
files = []
hgrepos = []
issue_num = 29714
keywords = ['3.6regression']
message_count = 10.0
messages = ['288912', '288916', '289065', '289085', '289086', '289087', '289092', '289106', '290290', '290291']
nosy_count = 4.0
nosy_names = ['vstinner', 'serhiy.storchaka', 'xiang.zhang', 'Nick Huber']
pr_nums = ['499', '504', '703', '552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue29714'
versions = ['Python 3.6', 'Python 3.7']

[NickHuber]

Python 3.6.0 (default, Mar  3 2017, 00:15:36) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> b'a\x00%i' % 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: not all arguments converted during bytes formatting
>>> b'a%i' % 1
b'a1'
>>> b'a%i\x00' % 1
b'a1\x00'

On python3.5, this works in all the scenarios

Python 3.5.1 (default, Jan 14 2017, 03:58:20) 
[GCC 4.9.2] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> b'a\x00%i' % 1
b'a\x001'
>>> b'a%i' % 1
b'a1'
>>> b'a%i\x00' % 1
b'a1\x00'

[serhiy-storchaka]

Could this be due to using _PyBytesWriter?

[zhangyangyu]

The problem is now _PyBytes_FormatEx uses strchr to sniff %. It should use memchr instead.

[zhangyangyu]

Thanks for the report, Nick!

[vstinner]

I confirm that I introduced the regression with the commit fa7762e: "Issue bpo-25349: Optimize bytes % args using the new private _PyBytesWriter API". Sorry about that, I always forget that str*() functions should not be used in Python since we support embedded null bytes :-/ It seems like only Python 3.6.0 release is affected.

"git tag --contains fa7762e" also returns v3.5.2, but I don't understand since I only pushed my change to the 3.6 branch...

[vstinner]

Oh, and thank you for adding new unit tests ;-) Do we need similar unit tests for str%args and str.format()?

[zhangyangyu]

I think no. String is not affected now and its code uses related macros so I don't think it could suffer possible regression.

[vstinner]

Xiang Zhang added the comment:

I think no. String is not affected now and its code uses related macros so I don't think it could suffer possible regression.

Regressions are not something "expected", shit happens, all the time.
Unexpected corner cases are common :-)

[zhangyangyu]

New changeset df6d7b4 by Xiang Zhang in branch '3.6':
[3.6] bpo-29714: Fix a regression that bytes format may fail when containing zero bytes inside. (GH-504)
df6d7b4

[zhangyangyu]

New changeset b76ad51 by Xiang Zhang in branch 'master':
bpo-29714: Fix a regression that bytes format may fail when containing zero bytes inside. (GH-499)
b76ad51