datetime and time equality and hashing problematic

[cjw296]

Bug report

Bug description:

I'm sure folks will pull out reasoning behind this from the inscrutable section in PEP 495 but this behaviour is honestly extremely surprising at best and just flat out wrong on first glance:

>>> from datetime import time, datetime, date
>>> time(fold=1) == time(fold=0)
True
>>> hash(time(fold=1)) == hash(time(fold=0))
True
>>> datetime(2024, 10, 27, fold=1) == datetime(2024, 10, 27, fold=0)
True
>>> hash(datetime(2024, 10, 27, fold=1)) == hash(datetime(2024, 10, 27, fold=0))
True

Even when deliberately specifying as explicitly as possible two points in time which are absolutely not the same (these are two points during the UK DST backwards transition later this year):

>>> from zoneinfo import ZoneInfo
>>> tz = ZoneInfo('Europe/London')
>>> dt1 = datetime(2024, 10, 27, 1, 30, fold=0, tzinfo=tz)
>>> dt2 = datetime(2024, 10, 27, 1, 30, fold=1, tzinfo=tz)
>>> dt1 == dt2 
True
>>> hash(dt1) == hash(dt2)
True

This has really unpleasant implications for things like storing points in time in a dictionary:

>>> d = {}
>>> d[dt1] = 1
>>> d[dt2] = 2
>>> d[dt1]
2
>>> d[dt2]
2

The inverse of this issue is reported in #115845.

Yes, I know the timestamp can be used:

>>> dt1.timestamp() == dt2.timestamp()
False

...but not on time objects, and timestamp also brings the local machine's timezone into play:

>>> def fold_equal(*args, **kw):
...   return datetime(*args, **kw, fold=0).timestamp() == datetime(*args, **kw, fold=1).timestamp()
... 
>>> fold_equal(2024, 10, 27, 0, 30)
True
>>> fold_equal(2024, 10, 27, 1, 30)
False
>>> fold_equal(2024, 10, 27, 2, 30)
True
>>> fold_equal(2024, 10, 27, 0, 30, tzinfo=ZoneInfo('America/Chicago'))
True
>>> fold_equal(2024, 10, 27, 1, 30, tzinfo=ZoneInfo('America/Chicago'))
True
>>> fold_equal(2024, 10, 27, 2, 30, tzinfo=ZoneInfo('America/Chicago'))
True

Concretely, it would be a lot less confusing if:

• time objects, and datetime objects where tzinfo is None are equal and hashed the same only if all of their attributes including fold are the same.

• datetime objects where tzinfo is not None are equal and hashed the same only if they represent the exact same point in time.

CPython versions tested on:

3.12

Operating systems tested on:

No response

Linked PRs

• gh-116035: Document that both tzinfo and fold are ignored in comparisons if tzinfo is the same #116187
• [3.12] gh-116035: Document that both tzinfo and fold are ignored in comparisons if tzinfo is the same (GH-116187) #116216
• [3.11] gh-116035: Document that both tzinfo and fold are ignored in comparisons if tzinfo is the same (GH-116187) #116217

[cjw296]

A quick scan of @pganssle 's https://blog.ganssle.io/articles/2018/02/a-curious-case-datetimes.html documents that this happens but I didn't spot any justification for it...

[pganssle]

I don't know if I've ever actually gotten to the bottom of why this requirement exists, but I've been meaning to figure it out for years. I don't have time to track it down at the moment, but it was the result of a discussion between @abalkin and @tim-one, with the result documented here.

This is the message where Tim explains that you can't make two dates differing only by their fold have the hash and equality guarantees required, but it's in the context of a suggestion to calculate the hash as if fold=0, which presumably they were doing to solve some other problem.

I don't really understand the concern with time, though, since frankly it's a bit weird that time has a fold attribute at all, since there's no way for time objects to have ambiguous times (I mean, you could deliberately write a time zone library that does this, but in the real world the only time zones where you can get an offset without an associated date are ones with fixed offsets).

[pganssle]

Also I don't think this should be labeled as a "bug" — I don't know what the labels are used for, but if we were to change this behavior it would be breaking a thoroughly documented feature with a decently long public discussion history behind it. This is not a situation where no one thought about this, it's more that they thought about it and picked a specific set of trade-offs. If we want to change it, it would be because we decided that we would prefer a different set of trade-offs (and prefer them so much that it is worth making breaking changes to datetime).

Not saying that we definitely can't do it or anything, just that it's not really a "bug" (and it's certainly not getting backported to feature-frozen branches).

[tim-one]

Removed the "bug" label because it's working as designed and documented. Plus the chance that it will change is approximately 0 😉.

It's not PEP 495 particularly at work here, but a basic rule throughout datetime: objects overwhelmingly work in "naïve time", the details of which are primarily spelled out across various footnotes on tables of operations. It means that timezone info is overwhelmingly ignored, unless two aware datetime objects have two different tzinfo members. Then things like equality comparison effectively convert to UTC under the covers before the comparison occurs.

But if two datetime objects have the same tzinfo member (as in your second example),, comparison treats them as naïve datetimes. Everything related to time zone is ignored then. That was intended from the start, and has worked that way from the start (long before PEP 495 came along, although that PEP had to try to preserve backward compatibility too).

The intent has always been that if you want timezone-aware arithmetic (including comparison), you should convert to UTC first (a zone where "naïve arithmetic" is always the right thing to do), do the arithmetic, and convert back when you're done.

A consequence is that, ya, you "shouldn't" be using aware datetimes as dict keys - unless naïve comparison is what you want (which, BTW, usually is what I want).

@pganssle, as I recall, time objects have a fold attribute for a marginal reason: so you can use

datetime.combine(dt.date(), dt.time())

to get back an object identical in all respects to dt. Adding a fold attribute to date objects would have been even sillier 😉.

I don't believe any other use was envisioned for giving time a fold.

[cjw296]

Good to see you're alive and well @tim-one, it's been a while!

Removed the "bug" label because it's working as designed and documented

Just because something is working as designed and documented doesn't mean there isn't a bug in either, in this case it would appear to be the design.

Plus the chance that it will change is approximately 0 😉

I don't doubt it, but that also doesn't mean it's not a bug ;-)

It means that timezone info is overwhelmingly ignored, unless two aware datetime objects have two different tzinfo members

Yeah, I mean this appears to be the crux of the behaviour problem. I think it's somewhat defensible, although I still view it as a design failure, that "folds are completely ignored when there's no tzinfo".

But I honestly can't see how this can be seen as anything but a bug:

>>> t1 = datetime(2024, 10, 27, 1, fold=0, tzinfo=ZoneInfo('Europe/London'))
>>> str(t1)
'2024-10-27 01:00:00+01:00'
>>> t2 = datetime(2024, 10, 27, 1, fold=1, tzinfo=ZoneInfo('Europe/London'))
>>> str(t2)
'2024-10-27 01:00:00+00:00'
>>> t1 == t2
True

These are fundamentally two different points in time, in what way can they be defended as being equal? :-)

The intent has always been that if you want timezone-aware arithmetic (including comparison), you should convert to UTC first (a zone where "naïve arithmetic" is always the right thing to do), do the arithmetic, and convert back when you're done.

A consequence is that, ya, you "shouldn't" be using aware datetimes as dict keys - unless naïve comparison is what you want (which, BTW, usually is what I want).

Where are these two things documented? They honestly feel like they should be in a .. warning:: at the top of https://docs.python.org/3/library/datetime.html

[pganssle]

These are fundamentally two different points in time, in what way can they be defended as being equal? :-)

It's because a lot of stuff around datetime handling is actually more context-dependent than you would think. They represent different absolute times, but the same civil time, because civil times with backwards offset changes have inherent ambiguities.

If you look at how the temporal library in Javascript has drawn their abstraction boundaries, you'll see why it is more complicated here — the datetime class collapses "exact times" and "civil times", and as Tim mentions, tends to actually be optimized for working with civil times. When you internalize this concept, everything about datetime makes a lot more sense.

If we had the option (and maybe we will one day), I'd probably build a new datetime module more along the lines of what temporal has done: with much cleaner separation between "instant in time" and "civil time".

So it's easily defensible why those things are equal: they both represent 01:00 on 2024-10-27 in London. The fact that 01:00 on 2024-10-27 happened twice and these represent the two different instances only matters if you are using a different concept of equality (specifically, the one where two datetimes are equal if they represent the same offset from epoch).

I also realized that yes, this particular fold "problem" is actually one that I understood already 😅 The one that I didn't fully grok yet is this one:

>>> from datetime import datetime, timedelta
>>> from zoneinfo import ZoneInfo
>>> dt1 = datetime(2024, 10, 27, 1, tzinfo=ZoneInfo("Europe/London"))
>>> dt1 == dt1.astimezone(ZoneInfo("America/New_York")
False
>>> dt2 = dt1 - timedelta(seconds=1)
>>> dt2 == dt2.astimezone(ZoneInfo("America/New_York"))
True

Any time changing fold would change the utcoffset() on either side of an equality sign, the datetimes are not considered equal, even if they represent the same time in UTC, despite the fact that that's normally how intra-zone comparisons work.

In that context, reading Tim's message in 2015 makes more sense, because if you are preserving hash/equality equivalence, either you need a situation where the hash depends on the fold, in which case equality must also depend on the fold for intra-zone comparisons or you need to do something like what they did in PEP 495, where the ambiguous datetimes don't compare equal to datetimes in other zones, and thus it's fine for them to have different hash values. Presumably reading through the original thread you can see why this particular set of trade-offs was made rather than another set. I would be surprised if it wasn't carefully considered, since years and years into my work with datetime I've still found stuff like this where I realize that datetime is actually very thoughtfully-designed, given the backwards compatibility constraints that exist and the original design intentions.

[tim-one]

But I honestly can't see how this can be seen as anything but a bug:

Your t1 and t2 have the same tzinfo member. Therefore every form of arithmetic (including comparisons) involving just them uses "naïve time". For example, not only do they compare equal, t1 - t2 returns timedelta(0). These things are all related.

It's what Guido intended, and is how it's always worked. You're not required to agree with the design, but fighting against such old and established decisions is almost certainly futile.

The docs could be clearer, and I've long thought they could be materially improved by adding a less formal introductory section pointing out the consequences of this design decision. They can be surprising. But then the very concept of "time zones" with changing offsets is an unprincipled political mess.

As a general thing, while the docs are pretty good about pointing out where tzinfo is ignored, those locations seemingly weren't updated to explicitly add that the later-added fold is also ignored in such cases.

For datetime __eq__, a footnote on the relevant table reads:

If both comparands are aware and have different tzinfo attributes, the comparison acts as comparands were first converted to UTC datetimes except that the implementation never overflows.

I could have sworn that at one time it also explicitly said that if both comparands are aware and have the same tzinfo attributes, comparison acts as if both comparands were naïve (as is done in the footnote for datetime.__sub__). It's weak to leave that implicit.

Paul wrote that datetime was "optimized" for "civil time", but it's more that Guido fundamentally only cared about "civil time". If you needed any form of time zone aware arithmetic, fine, convert to UTC, do naïve arithmetic there, and convert back.

Indeed, he cared so little about time zones that the original design deliberately omitted any way to disambiguate the "problem times" around DST transitions. I recall, at the time, his astonishment upon discovering that a C struct tm wasted an entire bit to store a tm_isdst flag, needed only for such disambiguation. He didn't want Python to do the same.

Which is the one original decision which eventually got reworked, to add fold.

I expect I'd agree with Paul about what would constitute a better design (stronger wall between naïve and aware). But in real life I don't have problems, because I don't fight the design: when I need aware arithmetic, I convert to UTC first.

>>> utc = ZoneInfo("utc")
>>> t2.astimezone(utc) == t1.astimezone(utc)
False
>>> print(t2.astimezone(utc) - t1.astimezone(utc))
1:00:00

[erlend-aasland]

Thanks for the explanations @tim-one and @pganssle. ISTM this should be retargeted as a docs issue.

[tim-one]

I could have sworn that at one time it also explicitly said that if both comparands are aware and have the same tzinfo attributes, comparison acts as if both comparands were naïve

Ya, it's seemingly random doc rot. I don't know when that info got lost, but at least the 3.8 docs were (to my eyes) quite clear about this:

If both comparands are aware, and have the same tzinfo attribute, the common tzinfo attribute is ignored and the base datetimes are compared. If both comparands are aware and have different tzinfo attributes, the comparands are first adjusted by subtracting their UTC offsets (obtained from self.utcoffset()).

Although everywhere the docs say (or said) tzinfo is ignored, they should add that fold is also ignored - or state that as a general rule.

EDIT: looks like the docs here dropped this important info starting with version 3.11. No idea why. Worse, they seem now to imply that equality does make time zone adjustments in all cases:

datetime objects are equal if they represent the same date and time, taking into account the time zone.

So I'll at least agree that the docs have become infested with bugs 😦.