http.client: CannotSendRequest after TimeoutError

[7cff]

Bug report

Bug description:

When using the low-level http.client API, connection timeouts cause the Connection object to enter an improper state which seemingly cannot be reset short of creating a new Connection object.

To reproduce, you'll need an address that produces a timeout. This can be done locally using Python's http.server and whichever firewall solution your OS has (since I'm on Linux, this means nftables):

table ip filter {
	chain input {
		type filter hook input priority filter;
		tcp dport 8000 drop
	}
}

python -m http.server

Finally, in a Python interpreter, do the following:

from http.client import HTTPConnection
conn = HTTPConnection('127.0.0.1', 8000)
# this request throws TimeoutError
conn.request('HEAD', '/')
# this request throws CannotSendRequest
conn.request('HEAD', '/')

The following do not fix the state of the Connection:

• getresponse(): Returns AttributeError
• connect(): Even if the later connection to the address works, the Connection remains in an invalid state

CPython versions tested on:

3.13

Operating systems tested on:

Linux

Linked PRs

• gh-141938/fix-http-client-state-reset #141941
• gh-141938: Fix HTTPConnection being stuck in a dirty state when HTTPResponse.begin() raises exceptions/errors #142047
• gh-141938: document treatment of OSError raised by HTTPConnection.getresponse #142339
• [3.14] gh-141938: document treatment of OSError raised by HTTPConnection.getresponse (GH-142339) #142704
• [3.13] gh-141938: document treatment of OSError raised by HTTPConnection.getresponse (GH-142339) #142705

[picnixz]

• What is the exact traceback about AttributeError and with the TimeoutError?
• What's the state of the connection class? (conn._HTTPConnection__state attribute) after getting the timeout error?
• I guess this can be reproduced without a custom iptable, say with a very short timeout and something that sleeps on the other side. Is it possible to produce such example instead to ease reproducer?

[TotalyEnglizLitrate]

• What is the exact traceback about AttributeError and with the TimeoutError?

>>> connection.getresponse()
Traceback (most recent call last):
  File "<python-input-3>", line 1, in <module>
    connection.getresponse()
    ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/nix/store/7gl4khq26h625mpqzkiiqsify3hclar1-python3-3.13.9/lib/python3.13/http/client.py", line 1430, in getresponse
    response.begin()
    ~~~~~~~~~~~~~~^^
  File "/nix/store/7gl4khq26h625mpqzkiiqsify3hclar1-python3-3.13.9/lib/python3.13/http/client.py", line 331, in begin
    version, status, reason = self._read_status()
                              ~~~~~~~~~~~~~~~~~^^
  File "/nix/store/7gl4khq26h625mpqzkiiqsify3hclar1-python3-3.13.9/lib/python3.13/http/client.py", line 292, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/nix/store/7gl4khq26h625mpqzkiiqsify3hclar1-python3-3.13.9/lib/python3.13/socket.py", line 719, in readinto
    return self._sock.recv_into(b)
           ~~~~~~~~~~~~~~~~~~~~^^^
TimeoutError: timed out
>>> connection.request('GET', '/')
Traceback (most recent call last):
  File "<python-input-4>", line 1, in <module>
    connection.request('GET', '/')
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/nix/store/7gl4khq26h625mpqzkiiqsify3hclar1-python3-3.13.9/lib/python3.13/http/client.py", line 1338, in request
    self._send_request(method, url, body, headers, encode_chunked)
    ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/7gl4khq26h625mpqzkiiqsify3hclar1-python3-3.13.9/lib/python3.13/http/client.py", line 1349, in _send_request
    self.putrequest(method, url, **skips)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/7gl4khq26h625mpqzkiiqsify3hclar1-python3-3.13.9/lib/python3.13/http/client.py", line 1175, in putrequest
    raise CannotSendRequest(self.__state)
http.client.CannotSendRequest: Request-sent

• What's the state of the connection class? (conn._HTTPConnection__state attribute) after getting the timeout error?

_CS_REQ_SENT i.e "Request-sent"

• I guess this can be reproduced without a custom iptable, say with a very short timeout and something that sleeps on the other side. Is it possible to produce such example instead to ease reproducer?

server

import http.server
import socketserver
import time

PORT = 8000

class RequestHandler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        time.sleep(5)

with socketserver.TCPServer(("", PORT), RequestHandler) as httpd:
    print(f"Serving at port {PORT}")
    httpd.serve_forever()

client

from http.client import HTTPConnection
connection = HTTPConnection('localhost', 8000, 1) # timeout less than sleep on server side
connection.request('GET', '/')
connection.getresponse() # raises TimeoutError
connection.request('GET', '/') # raises CannotSendRequest due to dirty state

Root cause

Looking at the HTTPConnection.getresponse method, the outer exception handler calls response.close() but not self.close(), so when response.begin() raises any exception other than ConnectionError, the connection state remains stuck in a dirty _CS_REQ_SENT state instead of being reset to _CS_IDLE even though the response was closed. To my knowledge, we should be doing self.close() here but I don't know if there are any edge cases where you would want to leave the connection open. some reading confirmed that we do want to close the connection here

cpython/Lib/http/client.py

Lines 1431 to 1453 in fa9519f

	try:
	try:
	if self.max_response_headers is None:
	response.begin()
	else:
	response.begin(_max_headers=self.max_response_headers)
	except ConnectionError:
	self.close()
	raise
	assert response.will_close != _UNKNOWN
	self.__state = _CS_IDLE
	if response.will_close:
	# this effectively passes the connection to the response
	self.close()
	else:
	# remember this, so we can tell when it is complete
	self.__response = response
	return response
	except:
	response.close()
	raise

[picnixz]

The docs actually say that only connection errors are handled automatically:

Changed in version 3.5: If a ConnectionError or subclass is raised, the HTTPConnection object will be ready to reconnect when a new request is sent.

So .. strictly speaking it's not really a bug I'd say. I don't know if changing the behavior would break other projects. Because we are now catching an OSError (I think TimeoutError inherits from it) and maybe the caller wants to propagate it.

[TotalyEnglizLitrate]

We are not catching the error, it is still being re-raised after response.close(), changing to self.close() would not affect propagation of TimeoutError or similar

[7cff]

I don't know if changing the behavior would break other projects. Because we are now catching an OSError (I think TimeoutError inherits from it) and maybe the caller wants to propagate it.

The currently linked PR simply calls close() (accounting for the EPIPE case) before reraising, so changing the behavior shouldn't break anything.

Even if it's decided that this isn't worth fixing, I think it's worth changing the documentation to make clear that the connection needs to be closed to fix the state machine, because this isn't mentioned currently and it did not occur to me that calling close() on a connection that was never opened in the first place would reset the state machine. In fact, the documentation about the connection being automatically reset is listed under getresponse(), when it's request() that's breaking the state machine and getresponse() as mentioned before doesn't work to fix the issue.

[picnixz]

In fact, the documentation about the connection being automatically reset is listed under getresponse(), when it's request() that's breaking the state machine and getresponse() as mentioned before doesn't work to fix the issue.

Yes, but the docs of getresponse which is just below also states:

Note that you must have read the whole response before you can send a new request to the server.

So calling request() afterwards without handling getresponse() possible exception is breaking the API contract. However, I agree that we should likely put that note under request() and not getresponse().

---

I think it's worth changing the documentation to make clear that the connection needs to be closed to fix the state machine, because this isn't mentioned currently and it did not occur to me that calling close() on a connection that was never opened in the first place would reset the state machine

I think it's good to do it.

---

We are not catching the error, it is still being re-raised after response.close(), changing to self.close() would not affect propagation of TimeoutError or similar

Again, I still don't know whether we want to close the entire connection in this case and not leave it to the caller. The caller may want to do something with the connection object before manually closing it. The docs clearly say that we reset the connection if we catch a ConnectionError (which is not the same as a TimeoutError) but it gives no guarantees on other kind of exceptions. Considering this behavior has been added after the initial implementation and is present for the past decades, we cannot just change this. Code that previously relied on the fact that the connection has not yet called close will be broken. Note that the underlying socket is also closed and the caller could do something else with that socket. So yes, it's a behavioral change. I don't know the reasons why it was designed like that though but I'd prefer having documenting this.

@pareshjoshij please avoid creating PRs when something has not yet been triaged.
@TotalyEnglizLitrate please avoid creating a PR when there is already one and also include tests in the future when changing a behavior or fixing something.

[7cff]

So calling request() afterwards without handling getresponse() possible exception is breaking the API contract.

I'm a bit confused; would you mind clarifying for me? In the case that getresponse() returns an Exception, what is supposed to be done to fulfill the contract? The docs don't say at the moment what to do once the error is caught, and we can't read the response as you mentioned because getting a response is returning an error.

I should also clarify that a TimeoutError can occur in two different scenarios:

• when a request is made against an existing socket (getresponse() returns TimeoutError)
• when a socket is being opened (request()/connect() returns TimeoutError)

The AttributeError occurs in the latter case:

Traceback (most recent call last):
  File "<python-input-44>", line 1, in <module>
    c.getresponse()
    ~~~~~~~~~~~~~^^
  File "/usr/lib/python3.13/http/client.py", line 1426, in getresponse
    response = self.response_class(self.sock, method=self._method)
  File "/usr/lib/python3.13/http/client.py", line 269, in __init__
    self.fp = sock.makefile("rb")
              ^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'makefile'

This is partly also why closing the connection didn't occur to me to reset the state machine: a connection was never opened in the first place because of the timeout, so there's no socket!

Hopefully this helps inform the documentation changes and planning for triaging this issue.