subprocess.Popen causes socket to remain open after close

[mamulsow]

BPO	3006
Nosy	@pitrou
Files	socketTest.py: socketTest.py socketinherit.py: Snippet that patches socket.socket to create uninheritable sockets

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2013-05-20.21:41:17.346>
created_at = <Date 2008-05-29.21:02:37.485>
labels = ['library', 'performance']
title = 'subprocess.Popen causes socket to remain open after close'
updated_at = <Date 2013-05-20.21:41:17.344>
user = 'https://bugs.python.org/mamulsow'

bugs.python.org fields:

activity = <Date 2013-05-20.21:41:17.344>
actor = 'neologix'
assignee = 'none'
closed = True
closed_date = <Date 2013-05-20.21:41:17.346>
closer = 'neologix'
components = ['Library (Lib)']
creation = <Date 2008-05-29.21:02:37.485>
creator = 'mamulsow'
dependencies = []
files = ['10471', '11029']
hgrepos = []
issue_num = 3006
keywords = []
message_count = 5.0
messages = ['67511', '70572', '71167', '126077', '189699']
nosy_count = 5.0
nosy_names = ['pitrou', 'ngrilly', 'kevinwatters', 'mamulsow', 'neologix']
pr_nums = []
priority = 'high'
resolution = 'out of date'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'resource usage'
url = 'https://bugs.python.org/issue3006'
versions = ['Python 2.7', 'Python 3.2', 'Python 3.3']

[mamulsow]

On Windows, when a suprocess.Popen command is issued while
a socket connection is being handled the socket connection
will not close until the output of the subprocess is consumed.
The connection remains open even though the request.close()
command returns successfully and the program starts listening
for the next connection.

On Windows, the request.close() call translates to the C function
closesocket. The closesocket function by default attempts to do
a graceful close in the background. By changing the linger structure,
the closesocket function can be made to do a hard close. That fixes
my problem, but then queued data may not be flushed before the
socket closes. I cannot figure out why the closesocket's graceful
shutdown is waiting for the Popen command to complete.

This problem does not show up with the equivalent os.popen command.

To reproduce:
run socketTest.py
use telnet to connect on port 6288
watch for connection to close or not

[kevinwatters]

I found a workaround for this issue (attached) via the kernel32.dll
function SetHandleInformation. You can patch the socket class to set all
newly created sockets as uninheritable.

It's not perfect--another thread could still spawn a subprocess in
between. We probably need some kind of API for setting socket inheritance.

[pitrou]

It's not perfect--another thread could still spawn a subprocess in
between

Then it could be done in socketmodule.c, when still holding the GIL.
Python code can change back the socket to inheritable afterwards if it
wants to.

We probably need some kind of API for setting socket inheritance.

Right.

[neologix]

I cannot figure out why the closesocket's graceful
shutdown is waiting for the Popen command to complete.

It doesn't. Your main process closes its socket. You could see it with a netstat/lsof (don't know under Windows).
The problem is that when you call subprocess.Popen(), there's a fork behind, and the child process receives a copy of the FD. That's why your socket is not deallocated until the subprocess completes (and closes the FD on exit).

This problem does not show up with the equivalent os.popen command.

That's because os.popen() is implemented as subprocess.Popen with close_fds=True, so the socket is closed before execve is called. IMHO, setting close_fds=True by default is the Right Thing to do (and I think it's now done by default in py3k to avoid races between concurrent popen calls, in addition to setting pipe's FD as FD_CLOEXEC).

[neologix]

It's fixed now that subprocess defaults to close_fds=True.