Test code coverage (C) job of Travis CI fails on test_distutils which creates _configtest.gcno file

[vstinner]

BPO	40237
Nosy	@brettcannon, @vstinner, @ammaraskar, @pganssle, @corona10, @shihai1991
PRs	bpo-40237: Fix the coverage gate in travis CI #19460

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-05-26.22:50:54.675>
created_at = <Date 2020-04-09.12:52:10.241>
labels = ['tests', '3.9']
title = 'Test code coverage (C) job of Travis CI fails on test_distutils which creates _configtest.gcno file'
updated_at = <Date 2021-05-26.22:50:54.675>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2021-05-26.22:50:54.675>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2021-05-26.22:50:54.675>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2020-04-09.12:52:10.241>
creator = 'vstinner'
dependencies = []
files = []
hgrepos = []
issue_num = 40237
keywords = ['patch']
message_count = 16.0
messages = ['366047', '366140', '366141', '366198', '366199', '366679', '366693', '366867', '366921', '366924', '368869', '368926', '371665', '371716', '371717', '394495']
nosy_count = 6.0
nosy_names = ['brett.cannon', 'vstinner', 'ammar2', 'p-ganssle', 'corona10', 'shihai1991']
pr_nums = ['19460']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue40237'
versions = ['Python 3.9']

[vstinner]

Example:
https://travis-ci.org/github/python/cpython/jobs/672698615

Output:
===========================================
0:09:01 load avg: 3.96 [104/421/1] test_distutils failed -- running: test_multiprocessing_forkserver (1 min 13 sec), test_unparse (6 min 21 sec)

Warning -- files was modified by test_distutils

Before: []

After: ['_configtest.gcno']

test test_distutils failed -- Traceback (most recent call last):

File "/home/travis/build/python/cpython/Lib/distutils/tests/test_build_ext.py", line 112, in test_build_ext

    assert_python_ok('-c', code)

File "/home/travis/build/python/cpython/Lib/test/support/script_helper.py", line 156, in assert_python_ok

return _assert_python(True, *args, **env_vars)

File "/home/travis/build/python/cpython/Lib/test/support/script_helper.py", line 142, in _assert_python

    res.fail(cmd_line)

File "/home/travis/build/python/cpython/Lib/test/support/script_helper.py", line 70, in fail

raise AssertionError("Process return code is %d\n"

AssertionError: Process return code is 1

command line: ['/home/travis/build/python/cpython/python', '-X', 'faulthandler', '-I', '-c', "\ntmp_dir = '/tmp/tmp3y4hpawx'\n\nimport sys\nimport unittest\nfrom test import support\n\nsys.path.insert(0, tmp_dir)\nimport xx\n\nclass Tests(unittest.TestCase):\n def test_xx(self):\n for attr in ('error', 'foo', 'new', 'roj'):\n self.assertTrue(hasattr(xx, attr))\n\n self.assertEqual(xx.foo(2, 5), 7)\n self.assertEqual(xx.foo(13,15), 28)\n self.assertEqual(xx.new().demo(), None)\n if support.HAVE_DOCSTRINGS:\n doc = 'This is a template module just for instruction.'\n self.assertEqual(xx.__doc__, doc)\n self.assertIsInstance(xx.Null(), xx.Null)\n self.assertIsInstance(xx.Str(), xx.Str)\n\n\nunittest.main()\n"]

stdout:

---

---

stderr:

---

Traceback (most recent call last):

File "<string>", line 9, in <module>

ImportError: /tmp/tmp3y4hpawx/xx.cpython-39-x86_64-linux-gnu.so: undefined symbol: __gcov_merge_add

---

===========================================

Python is built GCC with -ftest-coverage option. But it seems like this option is "leaked" to C flags used to build third party extensions in distutils.

Maybe CFLAGS_NODIST should be used instead of CFLAGS.

The Travis CI job runs these commands:
---
./configure
xvfb-run make -j4 coverage-report
make pythoninfo
bash <(curl -s https://codecov.io/bash) -y .github/codecov.yml
---

[shihai1991]

Maybe CFLAGS_NODIST should be used instead of CFLAGS.
Looks like it worked.

Python is built GCC with -ftest-coverage option.
More exact description is: Python is built GCC with --coverage option, --coverage option including -ftest-coverage option.

[shihai1991]

Looks like it worked.
Oh, sorry, I checked the wrong gate.

[shihai1991]

After CFLAGS replcaced by CFLAGS_NODIST, some extension module built failed, some info like:
*** WARNING: renaming "_struct" since importing it failed: build/lib.linux-x86_64-3.9/_struct.cpython-39-x86_64-linux-gnu.so: undefined symbol: __gcov_merge_add

the possible reason:
building '_struct' extension
gcc -pthread -fPIC -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers -Werror=implicit-function-declaration -fvisibility=hidden -O0 -pg --coverage -I./Include/internal -I./Include -I. -I/usr/local/include -I/temp/shihai/cpython/Include -I/temp/shihai/cpython -c /temp/shihai/cpython/Modules/_struct.c -o build/temp.linux-x86_64-3.9/temp/shihai/cpython/Modules/_struct.o
gcc -pthread -shared (xxxxxxxxxxlacking --coverage in here xxxxxxxxx) build/temp.linux-x86_64-3.9/temp/shihai/cpython/Modules/_struct.o -L/usr/local/lib -o build/lib.linux-x86_64-3.9/_struct.cpython-39-x86_64-linux-gnu.so

[shihai1991]

FWIW, gcc Instrumentation Options in https://gcc.gnu.org/onlinedocs/gcc/Instrumentation-Options.html

[vstinner]

I understand that with PR 19460, code coverage is not longer run on C extension modules of the stdlib. I'm not using C code coverage, so I don't know how it's supposed to be used or analyzed.

Hai: if you want to move on this issue, you have to find who uses this CI and how it should behave.

The other option is to only fix test_distutils.

[shihai1991]

Hai: if you want to move on this issue, you have to find who uses this CI and how it should behave.

Copy that. I found PR7773 updated the travis' coverage ci and a discussion issue in python/core-workflow#2, so I add brett and ammar to the noisy list, MAYBE they know the historical background.

[brettcannon]

The historical background is code coverage of C code seemed like a good idea, so we set it up. :) Not much else to it.

[shihai1991]

The historical background is code coverage of C code seemed like a good idea, so we set it up. :) Not much else to it.

Hm, code coverage of C code including two parts:

1. coverage of inner c code;
2. extension modules built by distutils;

If I understand correct, we force on 1) is good enough, no?

[vstinner]

Brett:

The historical background is code coverage of C code seemed like a good idea, so we set it up. :) Not much else to it.

If there is no user of code coverage *on pull requests*, maybe the simplest fix is to remove this job.

There is also Codecov service running somehow on Python branches through GitHub integration.

[vstinner]

It seems like results at published at: https://codecov.io/gh/python/cpython/

It seems like there is coverage on both C and Python code. I never used this service.