iter() not working with datetime.now()

Question

A simple snippet in Python 3.6.1:

import datetime
j = iter(datetime.datetime.now, None)
next(j)

returns:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

instead of printing out the classic now() behavior with each next().

I've seen similar code working in Python 3.3, am I missing something or has something changed in version 3.6.1?

Interesting, I'd expect this to work. It works in 3.4 and 3.5 too. — Martijn Pieters
– Martijn Pieters, Commented May 31, 2017 at 11:29
It works when you replace datetime.datetime.now with lambda: datetime.datetime.now() or partial(datetime.datetime.now). — Martijn Pieters
– Martijn Pieters, Commented May 31, 2017 at 11:31

Martijn Pieters · Accepted Answer · 2017-06-20 21:25:35Z

43

This is definitely a bug introduced in Python 3.6.0b1. The iter() implementation recently switched to using _PyObject_FastCall() (an optimisation, see issue 27128), and it must be this call that is breaking this.

The same issue arrises with other C classmethod methods backed by Argument Clinic parsing:

>>> from asyncio import Task
>>> Task.all_tasks()
set()
>>> next(iter(Task.all_tasks, None))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

If you need a work-around, wrap the callable in a functools.partial() object:

from functools import partial

j = iter(partial(datetime.datetime.now), None)

I filed issue 30524 -- iter(classmethod, sentinel) broken for Argument Clinic class methods? with the Python project. The fix for this has landed and is part of 3.6.2rc1.

edited Jun 20, 2017 at 21:25

answered May 31, 2017 at 11:52

Martijn Pieters

1.1m327 gold badges4.2k silver badges3.5k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

erip Over a year ago

+1, but as a point of interest, why would you prefer partial to something like, say, a lambda (like in your comment)?

Martijn Pieters Over a year ago

@erip: because it is faster. There is no Python frame to be created, the C implementation can call the datetime.datetime.now C function directly.

MSeifert · Accepted Answer · 2017-05-31 13:14:44Z

16

I assume you're using CPython and not another Python implementation. And I can reproduce the issue with CPython 3.6.1 (I don't have PyPy, Jython, IronPython, ... so I can't check these).

The offender in this case is the replacement of PyObject_Call with _PyObject_CallNoArg in the C equivalent of the callable_iterator.__next__ (your object is a callable_iterator) method.

The PyObject_Call does return a new datetime.datetime instance while _PyObject_CallNoArg returns NULL (which is roughly equivalent to an exception in Python).

Digging a bit through the CPython source code:

The _PyObject_CallNoArg is just a macro for _PyObject_FastCall which in turn is a macro for _PyObject_FastCallDict.

This _PyObject_FastCallDict function checks the type of the function (C-function or Python function or something else) and delegates to _PyCFunction_FastCallDict in this case because datetime.now is a C function.

Since datetime.datetime.now has the METH_FASTCALL flag it ends up in the fourth case but there _PyStack_UnpackDict returns NULL and the function is never even called.

I'll stop there and let the Python devs figure out what's wrong in there. @Martijn Pieters already filed a Bug report and they will fix it (I just hope they fix it soonish).

So it's a Bug they introduced in 3.6 and until it's fixed you need to make sure the method isn't a CFunction with the METH_FASTCALL flag. As workaround you can wrap it. Apart from the possibilities @Martijn Pieters mentioned there is also a simple:

def now():
    return datetime.datetime.now()

j = iter(now, None)
next(j)  # datetime.datetime(2017, 5, 31, 14, 23, 1, 95999)

edited May 31, 2017 at 13:14

answered May 31, 2017 at 12:22

MSeifert

155k41 gold badges357 silver badges380 bronze badges

2 Comments

Vidak Over a year ago

Great answer too, with added insight into the culprits. I'll keep the accepted mark on Martijn's answer as he's already submitted the issue with the Python project.

Martijn Pieters Over a year ago

It's a confirmed bug in the FASTCALL optimisations, and Victor Stinner has a preliminary patch out at github.com/python/cpython/pull/1886

Collectives™ on Stack Overflow

iter() not working with datetime.now()

2 Answers 2

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related