time.tzset does not reset _strptime's locale time cache

[mibanescu]

BPO	6478
Nosy	@abalkin, @bitdancer, @berkerpeksag, @serhiy-storchaka
Files	ttime.py: Reproducer for the problem. _strptime.py.patch issue6478.patch issue6478_v2.patch issue6478_v3.diff strptime_cache_timezone.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = <Date 2015-12-12.08:25:22.817>
created_at = <Date 2009-07-13.17:33:43.956>
labels = ['type-bug', 'library']
title = "time.tzset does not reset _strptime's locale time cache"
updated_at = <Date 2015-12-12.08:25:22.816>
user = 'https://bugs.python.org/mibanescu'

bugs.python.org fields:

activity = <Date 2015-12-12.08:25:22.816>
actor = 'serhiy.storchaka'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2015-12-12.08:25:22.817>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2009-07-13.17:33:43.956>
creator = 'mibanescu'
dependencies = []
files = ['14496', '17964', '27732', '27749', '29012', '41134']
hgrepos = []
issue_num = 6478
keywords = ['patch']
message_count = 15.0
messages = ['90497', '110086', '110091', '173879', '173880', '173905', '173941', '174824', '174826', '181711', '209738', '229171', '255146', '255835', '256244']
nosy_count = 6.0
nosy_names = ['belopolsky', 'r.david.murray', 'mibanescu', 'python-dev', 'berker.peksag', 'serhiy.storchaka']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue6478'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5', 'Python 3.6']

[mibanescu]

If one changes from one timezone to another within the same python
process, and if one tries to parse a time string that includes the
timezone, the library malfunctions.

See attached script for a reproducer.

The problem is that, even though time.tzset() is called, the LocaleTime
persisted in the TimeRE global is not reset. In my example, the EDT
timezone name, compiled from the previous TZ variable, is not valid
anymore in the 'Pacific/Fiji' timezone.

To witness the behavior, run the attached script with no arguments. It
will parse the time in the America/New_York timezone just fine.

Then run it with an argument (like python ttime.py 1). It will first
prime the _strptime cache in the Pacific/Fiji timezone, then attempt to
parse the same time string in the America/New_York timezone.

Finally, you can change the "if 0" to "if 1" for working around the problem.

This has been verified in 2.4.4 and 2.6.1 (did not have a chance to
verify it against python 2.6.2 yet).

[mibanescu]

Proposed patch attached.

[abalkin]

Thank you for the bug report and a patch.

This does look like a bug and proposed patch seems to be the simplest way to fix it. It is unfortunate that the entire TimeRE cache needs to be recalculated when only 'Z' entry is invalidated by TZ change. Please consider separating lang and TZ change checks and not reinitialize TimeRE cache object if only TZ changes, just replace the 'Z' entry.

The patch also needs unit tests. It should be straightforward to convert ttime.py into a test case. Please add a test case for both test_time and test_datetime. Note that in 3.2 datetime.strptime() will process '%Z' when '%z' is also present. For example,

>>> datetime.strptime('Fri Jul 25 13:26:29 EDT -0500 2008', '%a %b %d %H:%M:%S %Z %z %Y')
datetime.datetime(2008, 7, 25, 13, 26, 29, tzinfo=datetime.timezone(datetime.timedelta(-1, 68400), 'EDT'))

Please make sure to restore environment variables in test cleanup.

Also a nit on the implementation: cls is unused in _get_timezone(cls), so it would be more appropriate to make it staticmethod instead of classmethod. And tzset should probably be called inside _get_timezone.

[berkerpeksag]

Here is the second patch with Alexander Belopolsky's comments addressed.

[abalkin]

The patch looks good, but I have a few comments on the test:

1. Does it work on Windows? It seems to rely on Olson's TZ names.
2. Please use @run_with_tz decorator.

[berkerpeksag]

Thanks for the review.

1. Does it work on Windows? It seems to rely on Olson's TZ names.

Is time.tzset() function only available on Unix?

See: http://docs.python.org/dev/library/time.html#time.tzset

2. Please use @run_with_tz decorator.

I will update my patch to use @run_with_tz decorator.

[berkerpeksag]

2. Please use @run_with_tz decorator.

Done. Patch attached.

[berkerpeksag]

Alexander: Did you have a chance to review the test?

[abalkin]

issue6478_v2.patch looks good to me. There is a long line in _strptime.py which I will fix before committing.

[berkerpeksag]

There is a long line in _strptime.py which I will fix before
committing.

Ops, fixed.

Also, I've found a better place for the test in Lib/test/test_strptime.py
and moved it to the test_strptime.CacheTests.

When I applied the patch, test_bad_timezone test failed because of
this code in Lib/_strptime.py:

    @staticmethod
    def _get_timezone():
        try:
            time.tzset()
        except AttributeError:
            pass
        # ...

time.tzset() always resets time.tzname[1], so this part of the code
in test_bad_timezone[2] didn't work.

    tzname = time.tzname[0]
    time.tzname = (tzname, tzname)

I've fixed test_bad_timezone test by mocking the new
_strptime._TimeRE_cache.locale_time._get_timezone method.

[1] http://hg.python.org/cpython/file/default/Modules/timemodule.c#l827
[2] http://hg.python.org/cpython/file/default/Lib/test/test_strptime.py#l320

[serhiy-storchaka]

May be System Z Linux buildbot fails due to this bug (http://buildbot.python.org/all/builders/System%20Z%20Linux%203.x/builds/1154/steps/test/logs/stdio):

======================================================================
ERROR: test_strptime (test.datetimetester.TestDateTime_Pure)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-zlinux-z/build/Lib/test/datetimetester.py", line 1879, in test_strptime
    dt = strptime(dtstr, "%z %Z")
  File "/home/dje/cpython-buildarea/3.x.edelsohn-zlinux-z/build/Lib/datetime.py", line 1593, in strptime
    return _strptime._strptime_datetime(cls, date_string, format)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-zlinux-z/build/Lib/_strptime.py", line 500, in _strptime_datetime
    tt, fraction = _strptime(data_string, format)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-zlinux-z/build/Lib/_strptime.py", line 337, in _strptime
    (data_string, format))
ValueError: time data '-0500 EST' does not match format '%z %Z'

---

[bitdancer]

Berker: the patch doesn't apply cleanly any more. Also, about the test_bad_timezone modification...what about the previous check that the tzname wasn't in UTC/GMT? Isn't that still needed? Or perhaps better yet, an additional @run_with_tz decorator? (Note that I don't really *understand* this code, so I may be totally off base here... :)