patch to implement PEP 461 (%-interpolation for bytes)

[nascheme]

BPO	20284
Nosy	@nascheme, @ncoghlan, @vstinner, @ericvsmith, @ethanfurman, @vadmium
Files	pep-draft.txt: draft PEP (alternative to 460 & 461) 01-pep461.patch 02-code-a.patch 03-py2-flag.patch 04-py2-eq.patch issue20284.stoneleaf.01.patch issue20284.stoneleaf.tests_only.01.patch issue20284.stoneleaf.02.patch issue20284.stoneleaf.03.patch issue20284.stoneleaf.04.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ethanfurman'
closed_at = <Date 2015-01-29.23:26:31.652>
created_at = <Date 2014-01-16.21:33:09.311>
labels = ['interpreter-core', 'type-feature']
title = 'patch to implement PEP 461 (%-interpolation for bytes)'
updated_at = <Date 2015-01-29.23:26:31.651>
user = 'https://github.com/nascheme'

bugs.python.org fields:

activity = <Date 2015-01-29.23:26:31.651>
actor = 'ethan.furman'
assignee = 'ethan.furman'
closed = True
closed_date = <Date 2015-01-29.23:26:31.652>
closer = 'ethan.furman'
components = ['Interpreter Core']
creation = <Date 2014-01-16.21:33:09.311>
creator = 'nascheme'
dependencies = []
files = ['33507', '33574', '33575', '33576', '33577', '37616', '37617', '37703', '37745', '37766']
hgrepos = []
issue_num = 20284
keywords = ['patch', 'needs review']
message_count = 24.0
messages = ['208313', '208328', '208329', '208331', '208356', '208581', '215019', '224024', '233456', '233458', '233524', '233545', '233546', '233548', '233875', '234012', '234189', '234246', '234265', '234266', '234591', '234726', '234746', '234750']
nosy_count = 8.0
nosy_names = ['nascheme', 'ncoghlan', 'vstinner', 'eric.smith', 'Arfrever', 'ethan.furman', 'python-dev', 'martin.panter']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue20284'
versions = ['Python 3.5']

[nascheme]

This is a very rough, proof of concept patch that implements %-style formatting for bytes objects. Currently it calls __format__ with a bytes argument and expects a bytes result. I've only implemented
support for bytes formatting for the 'long' object.

Expected behavior:

>>> b'%s' % b'hello'
b'hello'
>>> b'%s' % 'hello'
TypeError is raised
>>> b'%s' % 123
b'123'
>>> b'%d' % 123
b'123'

Some issues:

• %s support is incomplete, needs to handle width and padding. I think it should be done in formatbytes().

• PyBytes_Format function very likely has bugs, I copied it mostly from Python 2 and quickly made it run, I did very little testing.

• long__format__ with a bytes argument is inefficient. It creates temporary Python objects that could be avoided. %-style formatting on bytes will be much less efficient than str in Python 2. We could inline the handling of certain types in PyBytes_Format, maybe longs only would be sufficent.

• I'm not sure overloading __format__ is the best approach. Maybe we should introduce a new method, say __ascii__ instead and have PyBytes_Format call that.

[nascheme]

I'm attaching v2 of my proposed patch. This one is quite a bit better, IMHO.

• Introduce __ascii__ as a special method, like __str__ but required to exist only if an ASCII-only format exists.

• Extract PyString_Format from Python 2.7 and update it for PyBytes.

• %c only accepts integers, not single character strs, maybe should
  accept length one byte objects.

• add %a, should be useful for debugging

• %s calls __bytes__ or __ascii__, otherwise gives a TypeError, should
  eventually support buffer API

• number formats work as they do in Python 2.

[vstinner]

I reviewed your second patch on Rietveld.

[nascheme]

Uploading new patch with the following changes:

• Allow length 1 bytes object as argument to %c.
• Make %r an alias for %a.

I will upload a draft PEP (proposed as a replacement for 461).

Victor, thanks for the review. My reply is:

• regarding duplicated code: almost all of the code I added came directly from Python 2.7. If it can but shared with unicode object, great. However, note that the Python 2.7 code is well tested and well optimized.

• regarding the introduction of __ascii__, see my draft PEP. PEP-461 cannot be implemented, adding __bytes__ to number objects breaks backwards compatibility.

• regarding the change to _datetime, I don't mind if it gets discarded but I think its useful behavior. Mostly I made it as a proof-of-concept.

[nascheme]

Another revision of the patch, now quite close to PEP-461 as proposed. Changes from PEP-461:

• include %a

• add -2 command-line flag. When enabled have %s fallback to calling PyObject_Str() and encoding to ASCII and also enable %r as alias for %a.

Changes from previous patch:

• remove __ascii__ special method, %s will only accept objects that
  implement __bytes__ or the buffer API, unless -2 command line is used

• use buffer API if available

• add -2 command-line option

• Add prototypes for PyBytes_Format and _PyUnicode_FormatLong

• improve some exception messages

Reference counting in PyBytes_Format is quite hairy, could use some review. The code is nearly the same as Python 2.x stringobject.c.

[nascheme]

I've updated my patch into a sequence, the first of which implements PEP-461.

02-code-a.patch adds support for %a (ascii() on arg)

03-py2-flag.patch makes %s and %r behave similar to Python 2 if a command
line flag is provided to the interpreter

04-py-eq.patch makes the command line flag also enable comparision between bytes() and str() (warning is generated).

[ethanfurman]

PEP-461 has been accepted. I'll look over the code soon.

[ncoghlan]

Just noting I'm working on some significant updates to the bytes and bytearray docs in bpo-21777. I'll try to get that ready for review and merged relatively soon, so the docs for this can build on top of those changes.

[vstinner]

Hi. I proposed twice to Ethan to implement the PEP-461, but he replied that he wants to implement it. So, what's the status of the implementation?

[vstinner]

I would be nice to share as much code as possible with the Unicode implementation. My idea was to add a "_PyBytesWriter" API, very close to the "_PyUnicodeWriter", to share code. Old patch implementing the _PyBytesWriter API: issue bpo-17742 (rejected because it was less efficient, the compiler produces less efficient machine code).