Regression for star argument parameter error messages

[kayhayen]

BPO	28257
Nosy	@rhettinger, @vstinner, @larryhastings, @ned-deily, @serhiy-storchaka, @serprex
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Dependencies	bpo-27358: BUILD_MAP_UNPACK_WITH_CALL is slow
Files	built_tuple_unpack_with_call.patch build-map-unpack-with-call-error-messages-3.5.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/serhiy-storchaka'
closed_at = <Date 2016-10-07.20:36:17.395>
created_at = <Date 2016-09-23.11:55:46.115>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'Regression for star argument parameter error messages'
updated_at = <Date 2017-03-31.16:36:25.077>
user = 'https://bugs.python.org/kayhayen'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:25.077>
actor = 'dstufft'
assignee = 'serhiy.storchaka'
closed = True
closed_date = <Date 2016-10-07.20:36:17.395>
closer = 'serhiy.storchaka'
components = ['Interpreter Core']
creation = <Date 2016-09-23.11:55:46.115>
creator = 'kayhayen'
dependencies = ['27358']
files = ['44856', '44924']
hgrepos = []
issue_num = 28257
keywords = ['patch']
message_count = 11.0
messages = ['277270', '277278', '277282', '277284', '277289', '277293', '277294', '277594', '277858', '277860', '278268']
nosy_count = 8.0
nosy_names = ['rhettinger', 'vstinner', 'larry', 'ned.deily', 'kayhayen', 'python-dev', 'serhiy.storchaka', 'Demur Rumed']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue28257'
versions = ['Python 3.5', 'Python 3.6', 'Python 3.7']

[kayhayen]

Hello,

there is a regression in the beta (alpha 4 was ok) for this kind of code:

print("Complex call with both invalid star list and star arguments:")

try:
a = 1
b = 2.0

    functionWithDefaults(1,c = 3,*a,**b)
except TypeError as e:
    print(repr(e))

try:
a = 1
b = 2.0

    functionWithDefaults(1,*a,**b)
except TypeError as e:
    print(repr(e))

try:
a = 1
b = 2.0

    functionWithDefaults(c = 1, *a,**b)
except TypeError as e:
    print(repr(e))

try:
a = 1
b = 2.0

    functionWithDefaults(*a,**b)
except TypeError as e:
    print(repr(e))

This prints with beta1 3.6

Complex call with both invalid star list and star arguments:
TypeError("'int' object is not iterable",)
TypeError("'int' object is not iterable",)
TypeError("'float' object is not iterable",)
TypeError('functionWithDefaults() argument after ** must be a mapping, not float',)

The later message is what they all probably should be like. This is 3.5 output:

Complex call with both invalid star list and star arguments:
TypeError('functionWithDefaults() argument after ** must be a mapping, not float',)
TypeError('functionWithDefaults() argument after ** must be a mapping, not float',)
TypeError('functionWithDefaults() argument after ** must be a mapping, not float',)
TypeError('functionWithDefaults() argument after ** must be a mapping, not float',)

The function itself doesn't matter obviously, it's never called. Please restore the old behavior, thanks.

Yours,
Kay

[serhiy-storchaka]

This is a consequence of bpo-27213. Actually there are two issues: with var-positional and var-keyword arguments.

But Python 3.5 is not consistent. It raises an exception with less detailed message if there are multiple var-positional or var-keyword arguments.

Var-positional arguments:

>>> f(*0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after * must be an iterable, not int
>>> f(1, *0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after * must be an iterable, not int
>>> f(*[], *0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable

Python 3.6 just raises the latter message in case of positional arguments and single var-positional argument.

>>> f(*0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after * must be an iterable, not int
>>> f(1, *0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
>>> f(*[], *0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable

This issue can't be fixed without adding new bytecode (BUILD_TUPLE_UNPACK_WITH_CALL). If it will be decided to fix it in 3.6, it may be worth to backport this to 3.5.

Var-keyword arguments:

Python 3.5:

>>> f(**[])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after ** must be a mapping, not list
>>> f(x=1, **0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after ** must be a mapping, not int
>>> f(x=1, **[])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after ** must be a mapping, not list
>>> f(**{}, **0)   
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
>>> f(**{}, **[])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'list' object is not a mapping

Python 3.6 raises less detailed error message in case of keyword arguments and single var-keyword argument.

>>> f(**0)                      
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after ** must be a mapping, not int
>>> f(**[])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after ** must be a mapping, not list
>>> f(x=1, **0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable
>>> f(x=1, **[])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'list' object is not a mapping
>>> f(**{}, **0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not a mapping
>>> f(**{}, **[])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'list' object is not a mapping

This issue can be fixed without changing bytecode. The patch faster_build_map_unpack_with_call.patch for bpo-27358 fixes it.

[serhiy-storchaka]

Ned and Larry.

Is it allowed to use the patch from bpo-27358 for fixing a regression in 3.6 and inconsistency in 3.5 for var-keyword arguments? The patch introduces new private function _PyDict_MergeEx() and touches the implementation of PyDict_Merge(). Maybe this can be fixed with smaller patch, but I don't know.

Is it allowed to add new opcode BUILD_TUPLE_UNPACK_WITH_CALL for fixing a regression in 3.6 and inconsistency in 3.5 for var-positional arguments? This is the only way. The bytecode already was changed in a bugfix release of 3.5 for more serious need: fixing potential segfault or security issue (bpo-27286).

[larryhastings]

It's too late to change this for 3.5.

[ned-deily]

That is a fairly big change to go in now but the error message regression is also big. With a review from a core developer and assuming no other objections, I think this can go into 360b2.

[rhettinger]

It's too late to change this for 3.5.

The whole point of having a beta release is to detect issues like this.

[ned-deily]

Raymond, Larry's comment was about 3.5, not 3.6. See my comment above.

[serhiy-storchaka]

Proposed patch fixes error message for var-positional arguments. It adds new opcode BUILD_TUPLE_UNPACK_WITH_CALL.

>>> min(1, *2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: min() argument after * must be an iterable, not int
>>> min(*[1], *2)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: min() argument after * must be an iterable, not int

[python-dev]

New changeset 88b319dfc909 by Serhiy Storchaka in branch '3.6':
Issue bpo-28257: Improved error message when pass a non-iterable as
https://hg.python.org/cpython/rev/88b319dfc909

New changeset bde594cd8369 by Serhiy Storchaka in branch 'default':
Issue bpo-28257: Improved error message when pass a non-iterable as
https://hg.python.org/cpython/rev/bde594cd8369

New changeset 40d7ce58ebd0 by Serhiy Storchaka in branch '3.5':
Issue bpo-28257: Backported a test.
https://hg.python.org/cpython/rev/40d7ce58ebd0

New changeset a8168a52a56f by Serhiy Storchaka in branch '2.7':
Issue bpo-28257: Backported a test.
https://hg.python.org/cpython/rev/a8168a52a56f

[serhiy-storchaka]

Here is a patch with smaller (in comparison with bpo-27358) change for 3.5 that improves error message when pass a non-mapping as second var-keyword argument.

Unpatched:

>>> f(**{'a': 1}, **[])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'list' object is not a mapping
>>> f(**{'a': 1}, **0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not iterable

Patched:

>>> f(**{'a': 1}, **[])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after ** must be a mapping, not list
>>> f(**{'a': 1}, **0)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: f() argument after ** must be a mapping, not int