More fixes for the Clinic mapping of converters to format units

[larryhastings]

BPO	24000
Nosy	@larryhastings, @zware, @serhiy-storchaka
Files	larry.one.more.clinic.format.unit.map.cleanup.1.txt larry.one.more.clinic.format.unit.map.cleanup.2.txt larry.one.more.clinic.format.unit.map.cleanup.3.txt larry.one.more.clinic.format.unit.map.cleanup.4.txt larry.one.more.clinic.format.unit.map.cleanup.5.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/larryhastings'
closed_at = <Date 2015-05-08.06:31:50.870>
created_at = <Date 2015-04-19.01:00:33.801>
labels = ['type-bug']
title = 'More fixes for the Clinic mapping of converters to format units'
updated_at = <Date 2015-05-08.06:31:50.868>
user = 'https://github.com/larryhastings'

bugs.python.org fields:

activity = <Date 2015-05-08.06:31:50.868>
actor = 'larry'
assignee = 'larry'
closed = True
closed_date = <Date 2015-05-08.06:31:50.870>
closer = 'larry'
components = []
creation = <Date 2015-04-19.01:00:33.801>
creator = 'larry'
dependencies = []
files = ['39113', '39115', '39125', '39288', '39289']
hgrepos = []
issue_num = 24000
keywords = []
message_count = 24.0
messages = ['241468', '241490', '241503', '241509', '241522', '241528', '241529', '241536', '241538', '241558', '242562', '242563', '242567', '242568', '242571', '242572', '242573', '242574', '242575', '242578', '242581', '242582', '242748', '242749']
nosy_count = 4.0
nosy_names = ['larry', 'python-dev', 'zach.ware', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue24000'
versions = ['Python 3.5']

[larryhastings]

I found another bug in the mapping of converters to format units. (s#, z#, and y# all allow zeroes.)

I've redone the approach for str_converter in an attempt to make it easier to read.

It'd be swell if, after this gets checked in (or rejected), somebody *else* took a pass to see if they could find any bugs.

[larryhastings]

Oh, heavens, yes, that's much nicer. Thanks for the suggestion, Serhiy!

[serhiy-storchaka]

Doesn't always zeroes == length?

[serhiy-storchaka]

Converters with encoding=True are not used in converted code. "es" is never used in still non-converted code and "et" is used only 6 times (for "idna" and "utf-8" encodings) and can be replaced with custom converter or inlined code. So I think the support of encoding=True should be removed from Argument Clinic soon.

Converters with length=True are used very rarely too, and they are mostly legacy converters. I think we should get rid of most of them in future.

[larryhastings]

Doesn't always zeroes == length?

"zeroes" requires "length", but "length" does not require "zeroes". As it happens all the format units supported by str always have both parameters either True or False. But the Py_UNICODE converter accepts "length" for format units u# and Z#, and doesn't support "zeroes".

I want the converters to accept common parameters. That will make comprehension easier for the casual reader. So I want to keep "length" as a separate parameter, even for str.

So I think the support of encoding=True should be removed
from Argument Clinic soon. [...] I think we should get rid
of [length] in future.

I disagree. My goal with Argument Clinic is that third-party developers will use it to write extensions. And we don't know how many extension modules are using es, es#, et, et#, s#, y#, z#, u#, and U#. I don't think we can remove any of this functionality until 4.0.

[serhiy-storchaka]

> Doesn't always zeroes == length?

"zeroes" requires "length", but "length" does not require "zeroes". As it happens all the format units supported by str always have both parameters either True or False. But the Py_UNICODE converter accepts "length" for format units u# and Z#, and doesn't support "zeroes".

"length=True" implies "zeroes=True" in the Py_UNICODE converter. "u" and "Z" don't allow null characters, "u#" and "Z#" allow null characters. "zeroes" doesn't add any information and isn't needed.

[serhiy-storchaka]

I disagree. My goal with Argument Clinic is that third-party developers will use it to write extensions. And we don't know how many extension modules are using es, es#, et, et#, s#, y#, z#, u#, and U#. I don't think we can remove any of this functionality until 4.0.

But we can deprecate them sooner.

[larryhastings]

"u#" and "Z#" allow null characters.

Not according to the documentation. 'u' explicitly says it does not allow NUL characters. 'Z', 'u#', and 'Z#' all say they are "variants" of 'u' but never mention that they might allow NUL characters.

[larryhastings]

New diff based on Serhiy's latest round of comments. Thanks, Serhiy! You are inexhaustable!

[serhiy-storchaka]

Not according to the documentation. 'u' explicitly says it does not allow NUL characters. 'Z', 'u#', and 'Z#' all say they are "variants" of 'u' but never mention that they might allow NUL characters.

I understand the note in "u" description as explicitly saying that "u#" allows null characters.

The documentation for format units needs an update for other reasons and there is an issue with ready patch for this.

[larryhastings]

Here's a freshened version of the patch. I updated the Clinic HOWTO.

Serhiy: You're right, length and zeroes always have the same value. Would you ever want length without allowing zeroes? Like, in the future, would we ever want
str(length=True)
so we're passed in the length but we still don't want to allow zeroes?

[serhiy-storchaka]

I would say it is very unlikely. In any case, if we have a pointer and a length, we always can check for zeros after parsing.

May be rename the str converter to pchar and the Py_UNICODE converter to pwchar? Usually the converter is named by C type, not Python type. "y" and "y#" even don't accept str.

[larryhastings]

I have no interest in naming "str" to "pchar".

Yes, *most* of the other converters are named after the C type they translate to. But so far converter names doesn't mention whether or not they represent pointers to types--it's "object", not "pobject", it's "PyBytesObject", not "pPyBytesObject". That suggests the name "char" for the converter. But we've already got a "char", and it would be confusing to use that one converter for both chars (small ints / single characters) and strings.

Adding "p" in front is not a convention we've ever used. The word "pchar" does not appear in the Python source tree. So the name "pchar" has no mnemonic value.

If I were to follow your advice, I should prefer the name "char_star". But now we're using nine letters for what is almost certainly the most common converter. And, again, the generic converter for objects is called "object", I do not propose to rename it to "object_star". So this converter's name would be an exception to the rule.

But then again, C strings themselves are an exception to the rule. They're not a built-in type as much as they are a *convention*. So any name we give it will ultimately be something of a compromise. And as compromises go "str" is great. So far nobody has been confused by it. It's short, and universally, instantly clear as to its meaning.

Furthermore, converters don't actually represent a C type. They represent a *mapping*, from a Python type (or types) to a single C type. So while it's a useful and productive convention to name converters after the type they convert to, it's hardly mandatory. And it would be a shame to squander clarity in service to a needless consistency.

p.s. If we hold ourselves to this firm ideal, where every converter is named after its C type, what should we call the "bool" converter? What should we call the "self" converter? What should you call your proposed "boolint" converter?