Regression test test_httpservers fails, hangs on Windows

[jeff5]

BPO	20155
Nosy	@terryjreedy, @orsenthil, @ezio-melotti, @bitdancer, @PCManticore, @jeff5
Files	issue20155_py.patch: Patch for 2.7

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-06-02.19:28:49.194>
created_at = <Date 2014-01-07.00:19:11.362>
labels = ['tests', 'type-bug', 'library', 'OS-windows']
title = 'Regression test test_httpservers fails, hangs on Windows'
updated_at = <Date 2015-06-02.19:28:49.192>
user = 'https://github.com/jeff5'

bugs.python.org fields:

activity = <Date 2015-06-02.19:28:49.192>
actor = 'r.david.murray'
assignee = 'none'
closed = True
closed_date = <Date 2015-06-02.19:28:49.194>
closer = 'r.david.murray'
components = ['Library (Lib)', 'Tests', 'Windows']
creation = <Date 2014-01-07.00:19:11.362>
creator = 'jeff.allen'
dependencies = []
files = ['34202']
hgrepos = []
issue_num = 20155
keywords = ['patch']
message_count = 26.0
messages = ['207497', '210926', '211990', '211999', '212004', '212026', '212652', '212754', '212755', '221458', '221498', '221499', '221803', '221858', '221859', '221860', '221862', '221863', '221866', '221893', '229576', '229580', '229661', '229662', '229666', '244686']
nosy_count = 8.0
nosy_names = ['terry.reedy', 'orsenthil', 'fwierzbicki', 'ezio.melotti', 'r.david.murray', 'Claudiu.Popa', 'python-dev', 'jeff.allen']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue20155'
versions = ['Python 2.7', 'Python 3.3', 'Python 3.4']

[jeff5]

When I run:
start python -m test.test_httpservers

test_request_line_trimming reports ERROR, and the test hangs at test_version_none. If I run a copy of the test in which the latter test is skipped with @unittest.skipIf(sys.platform == "win32", "..."), the error report is:

======================================================================
ERROR: test_request_line_trimming (main.BaseHTTPServerTestCase)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "test_httpservers.py", line 122, in test_request_line_trimming
    res = self.con.getresponse()
  File "C:\Python33\lib\http\client.py", line 1131, in getresponse
    response.begin()
  File "C:\Python33\lib\http\client.py", line 354, in begin
    version, status, reason = self._read_status()
  File "C:\Python33\lib\http\client.py", line 316, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "C:\Python33\lib\socket.py", line 297, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [WinError 10054] An existing connection was forcibly closed by the remote host

This is essentially the same for Python 2.7.6 and for the current development tip of Jython 2.7b1+, which is actually where the problem first manifested.

My machine is running 64-bit Windows 7 SP1, recently re-installed to a new, empty disk.

Careful testing, elaborating the failing tests, shows that what is sent in PUT and GET operations is not quite what is received. Something tampers with the connection between the client and the server. (Identical traffic where the verbs are not PUT and GET arrives as sent.) Something fiddles with the forward message, for example "correcting" the spurious \n in test_request_line_trimming to a full \r\n, and holding back the payload of a PUT even when it was in the first packet. On the reverse path, it appears to act on the error response itself by closing the connection, without passing it to the client.

Disabling the firewall (Windows Firewall and a commercial one), with the network cable unplugged, makes no difference. Nor does stopping anti-virus, anti-phishing, parental controls, etc.. However, stopping the Windows Basic Filtering Engine (BFE), makes the regression test run without error. Stopping the BFE takes out several dependent services, including Windows Firewall, but it seems likely the BFE itself is the culprit.

Although the cause lies in the platform, not in Python, it seems to me still an "issue" for Python that the tests fail on a common platform practically out of the box. I'll work on this in the context of the Jython test and report back here.

[terryjreedy]

Installed 3.4.0c1 on Windows:
test test_httpservers failed -- Traceback (most recent call last):
  File "C:\Programs\Python34.32\lib\test\test_httpservers.py", line 309, in test_invalid_requests
    self.check_status_and_reason(response, 501)
  File "C:\Programs\Python34.32\lib\test\test_httpservers.py", line 264, in check_status_and_reason
    self.assertEqual(response.status, status)
AssertionError: 200 != 501

I know very little about http server stuff.

[PCManticore]

Terry, I had the same problem with that failing test_httpservers.test_invalid_request using the latest build on Windows.
After debugging, I found out that the problem was caused by my antivirus solution. Its http scanning engine caught malformed http requests, like the one in the test ("gEt / hTTP/1.0\r\nhost: 127.0.0.1:50340\r\nAccept-Encoding: identity\r\n\r\n"; for instance) and it modified the first line by uppercasing it, thus making the perfect condition for a failing test.

[jeff5]

Thanks for adding to the evidence here. As discussed above, disabling the security product (which is Bitdefender) on my PC didn't stop the problem for me, and I'm reluctant to uninstall. I narrowed it to the Windows Base Filtering Engine, but perhaps the behaviour of the BFE is extended by installing BD.

If so, you could say this is not a Python problem, it is caused by BD "normalising" the HTTP. Or BD could say it is caused by expecting a defined result from abnormal HTTP.

I took the view it were best fixed at our end. I found I could test the same thing (AFAICT), but modify the tests so they don't get interfered with.
http://bugs.jython.org/issue2109
http://hg.python.org/jython/rev/6441fcfd940b

Would a patch made from this be applicable to CPython?

[terryjreedy]

Except where specifically indicated otherwise, the Python test suite should be the same, and correct, for all implementations. If that change is correct for Jython, it should be correct for CPython.

David, Ezio, or Senthil: does the change in the Jython patch look correct to any of you, enough that we should apply it?

[jeff5]

Actual patch for your convenience. I'm not set up to build CPython from source, so I've tested this with my installed CPython 2.7.6, and it's clean.

[As for keeping the tests in sync, yes that's our aim. Jython's Lib contains only the customised versions, and everything else comes from a copy of CPython's in lib-python/2.7. I'm always looking for a chance to delete one (i.e. use the common file).]

[bitdancer]

The second test looks correct to me. I don't understand what tset_request_line_trimming is testing (I haven't investigated), so I don't know if that one is correct, but it is certainly plausible.

[jeff5]

I worked out that the essence of the test is to insert an extra \n at the end of a GET request line. The request is syntactically invalid for HTTP. The \n\r\n appears like two blank lines, implying no headers, but the headers then follow where no data should be. The server is supposed to respond with status 501, because it does not, in fact, define a GET operation.

To find the replacement test plausible you have to accept that, with a server that doesn't define GET, the verb may as well be XYZBOGUS. Since the security filter doesn't understand that verb either (unlike GET), it doesn't interfere in the test.

[bitdancer]

Ah, OK. I was thinking that BaseHTTPRequestHandler defined GET, but I see that it doesn't.

So yes, this patch looks fine to me.

[PCManticore]

It would be nice if this could be committed. It's cumbersome to always have to deactivate the AV solution when running the tests on Windows, in order to avoid a failure of test_httpservers.

[python-dev]

New changeset ffdd2d0b0049 by R David Murray in branch '3.4':
bpo-20155: use fake HTTP method names so windows doesn't hang the tests.
http://hg.python.org/cpython/rev/ffdd2d0b0049

New changeset e67ad57eed26 by R David Murray in branch 'default':
merge: bpo-20155: use fake HTTP method names so windows doesn't hang the tests.
http://hg.python.org/cpython/rev/e67ad57eed26

New changeset b0526da56c54 by R David Murray in branch '2.7':
bpo-20155: use fake HTTP method names so windows doesn't hang the tests.
http://hg.python.org/cpython/rev/b0526da56c54