threading.Timer object is affected by changes to system time: Python locks should use a monotonic clock if available

[winfr34k]

BPO	31267
Nosy	@vstinner, @winfr34k, @anikey-m
Superseder	bpo-12822: threading.Condition.wait(timeout) should use a monotonic clock: use pthread_condattr_setclock(CLOCK_MONOTONIC)
Files	timer_testcase.py test_monotonic.py

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2019-01-17.10:04:50.532>
created_at = <Date 2017-08-23.20:47:35.939>
labels = ['type-bug', 'library']
title = 'threading.Timer object is affected by changes to system time: Python locks should use a monotonic clock if available'
updated_at = <Date 2021-10-01.08:49:02.732>
user = 'https://github.com/winfr34k'

bugs.python.org fields:

activity = <Date 2021-10-01.08:49:02.732>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2019-01-17.10:04:50.532>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2017-08-23.20:47:35.939>
creator = 'winfreak'
dependencies = []
files = ['47096', '47097']
hgrepos = []
issue_num = 31267
keywords = []
message_count = 8.0
messages = ['300765', '302249', '302257', '302259', '303893', '303934', '333798', '333840']
nosy_count = 4.0
nosy_names = ['vstinner', 'winfreak', 'Matthias Schmidt', 'anikey']
pr_nums = []
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '12822'
type = 'behavior'
url = 'https://bugs.python.org/issue31267'
versions = ['Python 3.5']

[winfr34k]

Hi,

I have been playing around with threading.Timer objects as timeouts for a project and noticed that my timeouts are affected by system time changes.

To test this, I have written a small demonstration script (timer_testcase.py) which you can find in the attachments. I would expect that after 120 seconds, a "Hello!" message will appear on the screen regardless of system time changes.

If you run the script like you would normally, you will see that it will work properly and my expectations are met. Now, run it again and immediately use "date +%T -s "HH:MM:SS"" where the time is >= 2 mins in the future. You will notice that the timer will latch immediately instead of waiting those 120 seconds before latching.

I have read Lib/threading.py to a certain extent and it seems like Timer objects are using monotonic time already because they use Events which use Conditions themselves, which references a "_timer" function that is just an alias for time.monotonic as one can see at the top of the file.
Then I checked out if the monotonic time works as expected (test_monotonic.py) by just jumping back and forth "in time", everything seemed to be normal.

Am I making a mistake and if so, where?

Thanks for any of your answers.

--
Best regards
Thomas

Environment: I'm using Python 3.5.3 on Debian 9.1 "Stretch" on x86_64.

[MatthiasSchmidt]

Hi,

any news on this issue? We are facing this one as well and looking forward for a fix/solution.

Cheers,

Matthis Schmidt

[vstinner]

threading.Timer is implemented with threading.Event.wait(timeout) which is implemented with threading.Condition.wait(timeout).

threading.Condition.wait(timeout) creates a lock called "waiter" and uses it to implement the wait:

   waiter.acquire(True, timeout)

So at the end of the chain, you find a lock created by _thread.allocate_lock() and the Lock.acquire(True, timeout) call.

At the C level, a lock is created by PyThread_allocate_lock(). The implementation of PyThread_allocate_lock() depends on the platform. Check:

>>> sys.thread_info
sys.thread_info(name='pthread', lock='semaphore', version='NPTL 2.25')

So in my case (Fedora 25), a Python lock is implemented as a semaphore:

• create the lock: sem_init()
• acquire the lock with a timeout: sem_timedwait(thelock, &ts)

The problem is that the sem_timedwait() function of the glibc doesn't allow to specify which clock is used:

https://sourceware.org/bugzilla/show_bug.cgi?id=14717

The second problem is that the glibc relies on the Linux kernel, and the kernel doesn't support specifiying a clock (extract of the glib bug):

"It seems this would need kernel work"

--

For sys.thread_info.lock == "mutex+cond", PyThread_allocate_lock(timeout) is implemented with pthread_mutex_lock() + pthread_cond_timedwait(). The good news is that this API allows to specify the clock:

pthread_condattr_init(&attr);
pthread_condattr_setclock(&attr, CLOCK_MONOTONIC);
pthread_cond_init(&cond, &attr);

... I already created bpo-23428 to call "pthread_condattr_setclock(&attr, CLOCK_MONOTONIC);" in Python, it was 2 years ago ;-)

[vstinner]

... I already created bpo-23428 to call "pthread_condattr_setclock(&attr, CLOCK_MONOTONIC);" in Python, it was 2 years ago ;-)

See also bpo-12822: "NewGIL should use CLOCK_MONOTONIC if possible".

[winfr34k]

Hello Victor,

thank you for your update on this issue.

By looking through the other bug reports you listed, it looks as if measures were implemented but never merged. Am I right with this observation? If so, will we ever see a switch over to monotonic time?

Thanks for the info :)

--
Best regards
Thomas

[vstinner]

it looks as if measures were implemented but never merged.

The blocker issue is that sem_timedwait() doesn't support CLOCK_MONOTONIC. The glibc has to be enhanced to support this new feature.

[vstinner]

I'm sorrry, I read the issue too quickly and misunderstood it. I guess that it's a duplicate of bpo-23428: "Use the monotonic clock for thread conditions on POSIX platforms". This issue is blocked the libc...

[vstinner]

I'm sorrry, I read the issue too quickly and misunderstood it. I guess that it's a duplicate of bpo-23428: "Use the monotonic clock for thread conditions on POSIX platforms". This issue is blocked the libc...

Oops, I wanted to post this comment on bpo-35747... There are too many duplicates of bpo-23428...

I close this issue as a duplicate of bpo-23428.