test_asyncio: relax timings even more

[skrah]

BPO	20336
Nosy	@gvanrossum, @jcea, @ncoghlan, @vstinner, @skrah, @1st1

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-06-06.13:15:10.964>
created_at = <Date 2014-01-21.22:29:51.627>
labels = ['invalid', 'type-bug', 'tests', 'expert-asyncio']
title = 'test_asyncio: relax timings even more'
updated_at = <Date 2014-06-06.13:38:03.524>
user = 'https://github.com/skrah'

bugs.python.org fields:

activity = <Date 2014-06-06.13:38:03.524>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2014-06-06.13:15:10.964>
closer = 'skrah'
components = ['Tests', 'asyncio']
creation = <Date 2014-01-21.22:29:51.627>
creator = 'skrah'
dependencies = []
files = []
hgrepos = []
issue_num = 20336
keywords = ['buildbot']
message_count = 10.0
messages = ['208714', '208745', '208906', '208909', '208911', '208912', '208913', '208939', '219882', '219883']
nosy_count = 6.0
nosy_names = ['gvanrossum', 'jcea', 'ncoghlan', 'vstinner', 'skrah', 'yselivanov']
pr_nums = []
priority = 'normal'
resolution = 'not a bug'
stage = None
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20336'
versions = ['Python 3.4']

[skrah]

It seems that the relaxed timings in some tests are still not
sufficient:

http://buildbot.python.org/all/builders/AMD64%20OpenIndiana%203.x/builds/7417/steps/test/logs/stdio

======================================================================
FAIL: test_time_and_call_at (test.test_asyncio.test_base_events.BaseEventLoopTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/export/home/buildbot/64bits/3.x.cea-indiana-amd64/build/Lib/test/test_asyncio/test_base_events.py", line 127, in test_time_and_call_at
    self.assertTrue(0.09 <= t1-t0 <= 0.9, t1-t0)
AssertionError: False is not true : 2.06036564335227

[gvanrossum]

I won't have time to fix this (and bardly to even review a fix) but this seems a straightforward fix and you can go ahead without me -- maybe you can get someone else with some asyncio experience to review your patch (e.g. Antoine or Victor).

[skrah]

I'm seeing other unrelated timeouts on the OpenIndiana machines.
Jesús, are the machines perhaps operating under an unusually high load?

[vstinner]

I'm seeing other unrelated timeouts on the OpenIndiana machines.

I read somewhere that the host of these virtual machines is *highly* loaded (load > 6). It's more a buildbot configuration issue than a test_asyncio issue.

[skrah]

I see. That's astonishing, because usually the builds are ultra-fast
on those machines.

[vstinner]

Tests failing because of timings is really annoying. It would be nice to add something in test.support to configure timings. A function can maybe wait until the system load is lower than a threshold, or we should add a parameter to configure a tolerance between the perfect timing and the expected timing.

The tolerance can be a simple factor, a floating point number >= 1.0.

For example, replace:
0.09 <= t1-t0 <= 0.9, t1-t0
with
0.1 - 0.01 * factor <= t1-t0 <= 0.1 + 0.8 * factor
or a new test function:
check_timing(t1-t0, 0.1, 0.01, 0.8)

[skrah]

+1 for doing something about timing failures. Perhaps also a
decorator @skip_if_load_higher_than(x).

[Nick, I'm adding you because our current test suite could lead
to "commit impossible" situations with Zuul.]

[ncoghlan]

I have a draft PEP for the Zuul idea, and yes, it notes the increased consequences of declaring a buildbot stable when it sometimes isn't. There's a reason OpenStack treats third party test results as advisory only rather than as gating criteria :P

[skrah]

Let's open a new issue for system load detection. This one is not asyncio specific.

[vstinner]

Let's open a new issue for system load detection. This one is not asyncio specific.

I opened issues bpo-20910 and bpo-20964 for example.