Skip to content

make tests more robust against running headless#2016

Merged
boegel merged 1 commit intoeasybuilders:developfrom
boegel:headless_tests
Nov 23, 2016
Merged

make tests more robust against running headless#2016
boegel merged 1 commit intoeasybuilders:developfrom
boegel:headless_tests

Conversation

@boegel
Copy link
Member

@boegel boegel commented Nov 22, 2016

For some reason unclear to me, a couple of the EasyBuild framework tests have been failing when running from my fork (https://github.com/boegel/easybuild-framework), with errors like this:

FAIL: Test multidiff function.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/boegel/easybuild-framework/test/framework/filetools.py", line 436, in test_multidiff
    self.assertEqual(lines[2], "3 %s- versionsuffix = '-test'%s (1/2) toy-0.0-gompi-1.3.12-test.eb" % (red, endcol))
  File "/home/travis/virtualenv/python2.6.9/lib/python2.6/site-packages/vsc_install-0.10.19-py2.6.egg/vsc/install/testing.py", line 79, in assertEqual
    raise AssertionError("%s:\nDIFF%s:\n%s" % (msg, limit, ''.join(diff[:self.ASSERT_MAX_DIFF])))
AssertionError: "3 \x1b[0;41m- versionsuffix = '-test'\x1b[0m (1/2) toy-0.0-gompi-1.3.12-test\x1b[0m..." != "3 \x1b[0;41m- versionsuffix = '-test'\x1b[0m (1/2) toy-0.0-gompi-1.3.12-test.eb":
DIFF:
- 3 - versionsuffix = '-test' (1/2) toy-0.0-gompi-1.3.12-test...?                                                                       ---- ^^
+ 3 - versionsuffix = '-test' (1/2) toy-0.0-gompi-1.3.12-test.eb
FAIL: Test det_terminal_size function.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/boegel/easybuild-framework/test/framework/systemtools.py", line 687, in test_det_terminal_size
    self.assertTrue(isinstance(height, int) and height > 0)
AssertionError

I'm not sure why they only fail for my fork and not the central repos, since the test setup is exactly the same. My best guess is that Travis is gradually rolling out some changes which have not yet hit all test environments.

This change makes the affected tests more robust against running headless. For the multidiff test, it makes some of the test cases slightly less strict, but that shouldn't affect the capability to catch regression (since the test is quite extensively).

@boegel boegel added this to the 3.0.1 milestone Nov 22, 2016
@boegel
Copy link
Member Author

boegel commented Nov 22, 2016

@wpoely86 please review

if isinstance(self.modtool, Lmod):
# GCC/4.6.3 is nowhere to be found (in $MODULEPATH)
load_err_msg = r"The following module\(s\) are unknown"
load_err_msg = r"The[\s\n]*following[\s\n]*module\(s\)[\s\n]*are[\s\n]*unknown"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is just weird?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why? Lmod is wrapping output lines differently in a headless (0-width) terminal, so the whole message doesn't fit on a single line anymore, it's wrapped.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It shouldn't be wrapping at all if the with is 0?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well, it does

remember, I'm also testing with old Lmod versions, so this is just something we need to deal with (it's also not a big issue imho)

we're not testing Lmod here, we're checking that some module is not available because EB didn't put it there

@boegel
Copy link
Member Author

boegel commented Nov 23, 2016

Thanks for the critical review @wpoely86!

@boegel boegel merged commit b8ab2f8 into easybuilders:develop Nov 23, 2016
@boegel boegel deleted the headless_tests branch November 23, 2016 12:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants