test_nntplib fails on CI

[serhiy-storchaka]

BPO	42794
Nosy	@malemburg, @pitrou, @vstinner, @tiran, @ned-deily, @serhiy-storchaka, @corona10, @miss-islington, @rrhodes
PRs	bpo-42794: Update test_nntplib to use offical group name for testing #24037 [3.9] bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) #24039 [3.8] bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) #24040 [3.7] bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) #24041 [3.6] bpo-42794: Update test_nntplib to use offical group name for testing (GH-24037) #24042 bpo-41994: Fix refcount issues in Python/import.c #22632
Files	openssl.cnf

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-01-01.19:25:55.606>
created_at = <Date 2020-12-31.09:36:38.503>
labels = ['3.8', 'tests', '3.9', '3.10']
title = 'test_nntplib fails on CI'
updated_at = <Date 2021-01-05.15:35:44.649>
user = 'https://github.com/serhiy-storchaka'

bugs.python.org fields:

activity = <Date 2021-01-05.15:35:44.649>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2021-01-01.19:25:55.606>
closer = 'vstinner'
components = ['Tests']
creation = <Date 2020-12-31.09:36:38.503>
creator = 'serhiy.storchaka'
dependencies = []
files = ['49714']
hgrepos = []
issue_num = 42794
keywords = ['patch']
message_count = 21.0
messages = ['384106', '384111', '384112', '384114', '384122', '384155', '384156', '384157', '384159', '384161', '384163', '384164', '384169', '384171', '384176', '384177', '384184', '384185', '384197', '384198', '384202']
nosy_count = 9.0
nosy_names = ['lemburg', 'pitrou', 'vstinner', 'christian.heimes', 'ned.deily', 'serhiy.storchaka', 'corona10', 'miss-islington', 'trrhodes']
pr_nums = ['24037', '24039', '24040', '24041', '24042', '22632']
priority = 'critical'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = None
url = 'https://bugs.python.org/issue42794'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[serhiy-storchaka]

Since yesterday ALL PRs are blocked by failing test_nntplib.

For example https://github.com/python/cpython/runs/1629664606?check_suite_focus=true:

======================================================================
ERROR: test_descriptions (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 250, in wrapped
    meth(self)
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 99, in test_descriptions
    desc = descs[self.GROUP_NAME]
KeyError: 'comp.lang.python'

======================================================================
FAIL: test_description (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 250, in wrapped
    meth(self)
  File "/home/runner/work/cpython/cpython/Lib/test/test_nntplib.py", line 85, in test_description
    self.assertIn("Python", desc)
AssertionError: 'Python' not found in ''

---

[rrhodes]

Server is returning '215 Newsgroup descriptions in form "group description"', but an empty list of lines, so it's reaching the 'nothing' case in _getdescriptions: https://github.com/python/cpython/blob/master/Lib/nntplib.py#L660

[vstinner]

Right now, it works for me:

$ ./python -m test -u all test_nntplib  -v -m test_descriptions
...
test_descriptions (test.test_nntplib.NNTPv1Tests) ... ok
test_descriptions (test.test_nntplib.NNTPv2Tests) ... ok
test_descriptions (test.test_nntplib.NetworkedNNTPTests) ... ok
skipped "<class 'test.test_nntplib.NetworkedNNTP_SSLTests'> got [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1122) connecting to 'nntp.aioe.org'"
...
Tests result: SUCCESS

[serhiy-storchaka]

On my computer tests are passed. They are only failed on CI.

[malemburg]

FWIW, I'm getting the same errors in PR #23140

Checking on the server that's being used, the newsgroup description is empty indeed: https://news.aioe.org/index.php?id=statistics-about-groups&group=comp.lang.python

[corona10]

== CPython 3.10.0a3+ (heads/master:c8a7b8fa1b, Jan 1 2021, 08:28:41) [GCC 7.5.0]
== Linux-5.4.0-1031-azure-x86_64-with-glibc2.27 little-endian
== cwd: /home/corona10/cpython/build/test_python_66899æ
== CPU count: 2
== encodings: locale=UTF-8, FS=utf-8
0:00:00 load avg: 0.00 Run tests sequentially
0:00:00 load avg: 0.00 [1/1] test_nntplib
test_descriptions (test.test_nntplib.NNTPv1Tests) ... ok
test_descriptions (test.test_nntplib.NNTPv2Tests) ... ok
test_descriptions (test.test_nntplib.NetworkedNNTPTests) ... ok
test_descriptions (test.test_nntplib.NetworkedNNTP_SSLTests) ... ERROR

======================================================================
ERROR: test_descriptions (test.test_nntplib.NetworkedNNTP_SSLTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/corona10/cpython/Lib/test/test_nntplib.py", line 250, in wrapped
    meth(self)
  File "/home/corona10/cpython/Lib/test/test_nntplib.py", line 99, in test_descriptions
    desc = descs[self.GROUP_NAME]
KeyError: 'comp.lang.python'

---

Ran 4 tests in 4.734s

FAILED (errors=1)
test test_nntplib failed
test_nntplib failed

== Tests result: FAILURE ==

1 test failed:
test_nntplib

Total duration: 4.8 sec
Tests result: FAILURE

My test machine success to reproduce this issue.

(Pdb) descs
{}

There might be 4 ways to solve this issue.

1. Re-create group and skip test while descs is empty
2. Update test to use other NNTP server which supports SSL

[corona10]

4 ways / 2 ways

[corona10]

I submit the patch to use group_name which is mentioned from https://news.aioe.org/manual/aioe-hierarchy/

IMHO, it will be not changed frequently than what we used before.

[serhiy-storchaka]

The test was passed on my computer because test.test_nntplib.NetworkedNNTP_SSLTests was skipped because of the configuration of SSL on Ubuntu 20.04 (see bpo-41561).

After using custom openssl.cnf I get the same errors.

$ OPENSSL_CONF=~/py/openssl.cnf ./python -m test -vuall test_nntplib -m 'test_description*'

The NNTP server nntp.aioe.org is used instead of snews.gmane.org since 2010 (45ca987). Sadly there is no reference to any discussion, maybe Antoine remember some details. What other alternatives were considered? How to handle similar breakage of external servers?

[corona10]

How to handle similar breakage of external servers?

For tests with the external environment,
How about skipping tests if servers are not available?
It is assumed that if there is a change related to NNTP, it is well checked in the CI and review.
This is because, at this point, reviewers and patch submitters check whether tests with external servers are passed.

The reason for testing with the outside is to confirm that the whole operation works well.
If external servers are not available, we can emit warning messages so that we can recognize when an issue occurs.

Other than that, I can think of local server testing or mocking testing, but I am not sure if it is easy way for the SSL environment.

[corona10]

For tests with the external environment,
How about skipping tests if servers are not available?

Hmm, I change my mind this is not a good idea :(