Skip to content

Regression tests on macOS sometimes randomly fail #702

@s3rvac

Description

@s3rvac

This macOS build has succeeded. The one after that, which is for a commit that only changed the changelog file, failed because of the following test:

bugs.selective-decompilation-strange-behaviour.TestDecompileSelect0x0DecodeAll (main.ex --sel[..])  [ FAIL ]  (1.15s)
F
======================================================================
FAIL: test_has_no_functions (bugs.selective-decompilation-strange-behaviour.TestDecompileSelect0x0DecodeAll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "retdec-regression-tests-framework/regression_tests/tools/decompiler_test.py", line 65, in setUp
    self.assertEqual(self.decompiler.return_code, 0, msg=msg)
AssertionError: 1 != 0 : decompiler failed; output:
...
Running phase: Simple types recovery optimization ( 0.65s )
Running phase: Stack pointer operations optimization ( 0.65s )
Running phase: Instruction idioms optimization ( 0.65s )
Running phase: LLVM ( 0.65s )
Running phase: LLVM instruction optimization ( 0.65s )
Running phase: Instruction idioms optimization ( 0.65s )
Running phase: Phi removal ( 0.65s )
Running phase: Value protection optimization ( 0.65s )
Error: Decompilation to LLVM IR failed

I restarted the build after that, and it failed again, but because of another test:

integration.palindrome.Test_2017 (palindrome.x86.clang.O0.g.elf)                                    [ FAIL ]  (0.85s)
F
======================================================================
FAIL: test_produce_expected_output (integration.palindrome.Test_2017)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "retdec-regression-tests/integration/palindrome/test.py", line 13, in test_produce_expected_output
    expected_output='krk is a palindrome\n'
  File "retdec-regression-tests-framework/regression_tests/tools/decompiler_test.py", line 97, in assert_c_produces_output_when_run
    self.assertEqual(actual_output, expected_output, msg)
AssertionError: 'Try again! (krk)\n' != 'krk is a palindrome\n'
- Try again! (krk)
+ krk is a palindrome
 : where the input was: krk

We either have issues with non-determinism in our tests on macOS, or there is something wrong with the macOS agent.

Possibly related issues: #267, #265, #264.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions