Add timespec optional flag to datetime isoformat() to choose the precision

[smontanaro]

BPO	19475
Nosy	@malemburg, @gvanrossum, @tim-one, @terryjreedy, @abalkin, @vstinner, @ezio-melotti, @berkerpeksag, @vadmium, @matrixise, @alessandrocucci
Files	issue19475.patch: Patch issue19475_v2.patch: Patch issue19475_v3.patch: Patch issue19475_v4.patch: Patch v4 issue19475_v5.patch: Patch v5 issue19475_v6.patch: Patch v6 issue19475_v7.patch: Patch v7 issue19475_v8.patch: Patch v8 issue19475_v9.patch: Patch v9 issue19475_v10_datetime_time.patch: added timespec to time.isoformat issue19475_v11.patch: Patch v11 issue19475_v12.patch issue19475_v13.patch: added milliseconds format to timespec issue19475_v14.patch: added milliseconds to documentation issue19475_v15.patch issue19475_v16.patch issue19475_v17.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = <Date 2016-03-06.19:58:58.041>
created_at = <Date 2013-11-01.17:05:21.528>
labels = ['extension-modules', 'easy', 'type-feature']
title = 'Add timespec optional flag to datetime isoformat() to choose the precision'
updated_at = <Date 2016-03-06.19:58:58.039>
user = 'https://github.com/smontanaro'

bugs.python.org fields:

activity = <Date 2016-03-06.19:58:58.039>
actor = 'python-dev'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2016-03-06.19:58:58.041>
closer = 'python-dev'
components = ['Extension Modules']
creation = <Date 2013-11-01.17:05:21.528>
creator = 'skip.montanaro'
dependencies = []
files = ['40018', '40024', '40117', '41329', '41371', '41381', '41387', '41388', '41391', '41420', '41448', '41464', '41988', '41994', '42007', '42031', '42063']
hgrepos = []
issue_num = 19475
keywords = ['patch', 'easy']
message_count = 94.0
messages = ['201917', '201918', '201922', '201924', '201925', '201926', '201929', '201931', '201932', '201934', '201935', '201943', '202220', '202238', '202239', '202242', '202243', '202270', '202274', '202276', '202282', '221958', '221959', '247265', '247401', '247402', '247439', '247723', '247948', '256458', '256470', '256475', '256476', '256477', '256478', '256479', '256480', '256481', '256483', '256486', '256513', '256536', '256552', '256556', '256560', '256561', '256591', '256764', '256813', '256855', '256858', '256866', '256967', '256991', '257196', '257202', '257203', '257284', '257288', '257534', '258289', '258473', '258477', '258479', '258480', '258481', '258482', '258483', '258485', '258493', '260629', '260645', '260695', '260725', '260875', '260876', '260878', '261060', '261066', '261068', '261073', '261074', '261077', '261078', '261080', '261081', '261082', '261083', '261084', '261085', '261086', '261133', '261135', '261269']
nosy_count = 14.0
nosy_names = ['lemburg', 'gvanrossum', 'tim.peters', 'terry.reedy', 'belopolsky', 'vstinner', 'ezio.melotti', 'cvrebert', 'python-dev', 'berker.peksag', 'martin.panter', 'matrixise', 'jerry.elmore', 'acucci']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue19475'
versions = ['Python 3.6']

[smontanaro]

I have a CSV file. Here are a few rows:

"2013-10-30 14:26:46.000528","1.36097023829"
"2013-10-30 14:26:46.999755","1.36097023829"
"2013-10-30 14:26:47.999308","1.36097023829"
"2013-10-30 14:26:49.002472","1.36097023829"
"2013-10-30 14:26:50","1.36097023829"
"2013-10-30 14:26:51.000549","1.36097023829"
"2013-10-30 14:26:51.999315","1.36097023829"
"2013-10-30 14:26:52.999703","1.36097023829"
"2013-10-30 14:26:53.999640","1.36097023829"
"2013-10-30 14:26:54.999139","1.36097023829"

I want to parse the strings in the first column as timestamps. I can, and often do, use dateutil.parser.parse(), but in situations like this where all the timestamps are of the same format, it can be incredibly slow. OTOH, there is no single format I can pass to datetime.datetime.strptime() that will parse all the above timestamps. Using "%Y-%m-%d %H:%M:%S" I get errors about the leftover microseconds. Using "%Y-%m-%d %H:%M:%S".%f" I get errors when I try to parse a timestamp which doesn't have microseconds.

Alas, it is datetime itself which is to blame for this problem. The above timestamps were all printed from an earlier Python program which just dumps the str() of a datetime object to its output CSV file. Consider:

>>> dt = dateutil.parser.parse("2013-10-30 14:26:50")
>>> print dt
2013-10-30 14:26:50
>>> dt2 = dateutil.parser.parse("2013-10-30 14:26:51.000549")
>>> print dt2
2013-10-30 14:26:51.000549

The same holds for isoformat():

>>> print dt.isoformat()
2013-10-30T14:26:50
>>> print dt2.isoformat()
2013-10-30T14:26:51.000549

Whatever happened to "be strict in what you send, but generous in what you receive"? If strptime() is going to complain the way it does, then str() should always generate a full timestamp, including microseconds. The above is from a Python 2.7 session, but I also confirmed that Python 3.3 behaves the same.

I've checked 2.7 and 3.3 in the Versions list, but I don't think it can be fixed there. Can the __str__ and isoformat methods of datetime (and time) objects be modified for 3.4 to always include the microseconds? Alternatively, can the %S format character be modified to consume optional decimal point and microseconds? I rate this as "easy" considering the easiest fix is to modify __str__ and isoformat, which seems unchallenging.

[ezio-melotti]

See bpo-7342.

[bitdancer]

It may be simple but as Ezio has pointed out, it has already been rejected :)

The problem with being generous in what you accept in this context is that the parsing is using a specific format string, and the semantics of that format string are based on external "standards" and are pretty inflexible.

The pythonic solution, IMO, is to have datetime's constructor accept what its str produces. And indeed, exactly this has been suggested by Alexander Belopolsky in bpo-15873. So I'm going to close this one as a duplicate of that one.

[smontanaro]

I don't accept your conclusion. I understand that making %S consume microseconds or ".%f" be "optional" would be a load. What's the problem with forcing __str__ and isoformat to emit microseconds in all cases though? That would allow you to parse what they produce using existing code. No new constructor needed.

The issue of sometimes emitting microseconds, sometimes not, is annoying, even beyond this issue. I think for consistency's sake it makes sense for the string version of datetime and time objects to always be the same length.

[bitdancer]

It's not my conclusion. It's Guido's and the other developers who designed datetime. Argue with them. (I'd guess it would be better argued on python-ideas rather than python-dev, but use your own judgement.)

[tim-one]

The decision to omit microseconds when 0 was a Guido pronouncement, back when datetime was first written. The idea is that str() is supposed to be friendly, and for the vast number of applications that don't use microseconds at all, it's unfriendly to shove ".000000" in their face all the time. Much the same reason is behind why, e.g., str(2.0) doesn't produce "2.0000000000000000".

I doubt this will change. If you want to use a single format, you could massage the data first, like

if '.' not in dt:
    dt += ".000000"

[smontanaro]

Okay, so no to __str__. What about isoformat?

[tim-one]

I don't know, Skip. Since .isoformat() and str() have always worked this way, and that was intentional, it's probably going to take a strong argument to change either.

[gvanrossum]

Well, I don't know if this sways anything, but I was probably responsible, and I think my argument was something about not all timestamp sources having microseconds, and not wanting to emit the ".000000" in that case. If I could go back I'd probably do something else; after all str(1.0) doesn't return '1' either. But that's water under the bridge; "fixing" this is undoubtedly going to break a lot of code.

Maybe we can give isoformat() a flag parameter to force the inclusion or exclusion of the microseconds (with a default of None meaning the current behavior)?

[smontanaro]

The ultimate culprit here is actually the csv module. :-) It calls str() on every element it's about to write. In my applications which write to CSV files I can special case datetime objects.

I will stop swimming upstream.