Create a new helper function that enable to test that an operation don't hang more than a given timeout.

[mouad]

BPO	12410
Nosy	@vstinner
Files	operation_timeout.patch: Add a helper method to make sure that an operation will not last more than a given timeout.

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-02-05.15:31:10.352>
created_at = <Date 2011-06-25.14:49:35.433>
labels = ['type-feature', 'tests']
title = "Create a new helper function that enable to test that an operation don't hang more than a given timeout."
updated_at = <Date 2012-02-05.15:31:10.351>
user = 'https://bugs.python.org/mouad'

bugs.python.org fields:

activity = <Date 2012-02-05.15:31:10.351>
actor = 'neologix'
assignee = 'none'
closed = True
closed_date = <Date 2012-02-05.15:31:10.352>
closer = 'neologix'
components = ['Tests']
creation = <Date 2011-06-25.14:49:35.433>
creator = 'mouad'
dependencies = []
files = ['22469']
hgrepos = []
issue_num = 12410
keywords = ['patch']
message_count = 6.0
messages = ['139075', '139091', '139099', '139123', '139127', '152692']
nosy_count = 3.0
nosy_names = ['vstinner', 'neologix', 'mouad']
pr_nums = []
priority = 'normal'
resolution = 'rejected'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue12410'
versions = ['Python 3.2']

[mouad]

While working on issue bpo-12157 [http://bugs.python.org/issue12157], I needed a function that make sure that an operation will not hang forever, for this reason i have create this helper function that support the context manager protocol and accept a timeout as an argument and raise an IOError if the operation didn't terminate before that timeout.

[neologix]

It's a little bit more complicated than that:

• signals and threads don't mix well together
• this will make syscalls fail with EINTR
• the old SIGALRM handler is lost
• etc

In short, don't use signals.
I'm not sure there's a reliable way to write such a general-purpose wrapper (usually one can use select() with a timeout or spawn a subprocess and use communicate's timeout to achieve this kind of things).
In your use case (issue bpo-12157), I think that letting the test block is fine, since:

• there's no easy way to add a timeout (but you could spawn a new interpreter and use communicate with a timeout if you really wanted to)
• it will be caught by the faulthandler module
• a test killed by faulthandler's timeout is more interesting to fix that a "common" failed test ;-)

[mouad]

Thanks for the instructive feedback :)

I totally agree i guess there is a lot of issues that i didn't think of :-(, my first thinking was to use "Pool.join" timeout argument but it was removed in 3.2 (by the way i didn't find the issue or the rational that lead to this change).

And now that i know about "faulthandler" module i guess that will make also my life easier :), i will rewrite the patch in the issue bpo-12157 to not use any *fancy* way to check if it will hang.

cheers,

[vstinner]

alarm() is one possible implementation, but Charles-François listed some drawbacks.

You can also use resource.setrlimit(RLIMIT_CPU), but the timeout is the CPU time (e.g. you cannot stop a sleep) and it is not portable (e.g. resource is not available on Windows).

Another possible implementation is a thread. faulthandler uses an "hidden" thread (implemented in C): a thread ignoring all signals using pthread_sigmask. Python threads are not reliable for a timeout because of the GIL, and it is not easy to "interrupt" another thread from the "timeout" thread. For example, you cannot (easily) raise an exception in another thread.

I'm not sure there's a reliable way to write such a general-purpose
wrapper

I agree, but it doesn't mean that it is not possible :-)

I think that you should try to implement in C a thread ignoring all signals. It becomes more complex when you have to implement the "interrupt the current thread" (current thread, or maybe the thread using the operation_timeout context manager?) part.

I suppose that you will have to use low-level "tricks" and you will have to experiment your tool on different platform.

You should start this project outside CPython (as a third party module), and then ask for an integration when your work is well tested. You have to know that a module "dies" when it enters CPython: you have to wait something like 18 months to modify it, so you have to be sure that your code is "correct" ;-)

[vstinner]

Oh, there is another possible implementation: use a subprocess. But if the timeout is implemented using a subprocess, the syntax cannot be:

with timeout(5):
   do_something()

It should be something like:

timeout(5, """if 1:
   import os, sys
   ...
   do_something()
   ...
   sys.exit(0)
""")

Some tests are already doing that manually.

[neologix]

Closing, since it's hard to write correctly, and apparently not that useful.