add a fallback socketpair() implementation to the socket module

[neologix]

BPO	18643
Nosy	@pitrou, @vstinner
Files	socketpair.diff sendall_write.diff socketpair-1.diff socketpair-3.diff socketpair-4.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2014-10-14.21:24:28.083>
created_at = <Date 2013-08-03.12:11:35.287>
labels = ['type-feature', 'library']
title = 'add a fallback socketpair() implementation to the socket module'
updated_at = <Date 2014-10-14.21:24:28.082>
user = 'https://bugs.python.org/neologix'

bugs.python.org fields:

activity = <Date 2014-10-14.21:24:28.082>
actor = 'neologix'
assignee = 'none'
closed = True
closed_date = <Date 2014-10-14.21:24:28.083>
closer = 'neologix'
components = ['Library (Lib)']
creation = <Date 2013-08-03.12:11:35.287>
creator = 'neologix'
dependencies = []
files = ['31452', '31458', '32909', '36050', '36055']
hgrepos = []
issue_num = 18643
keywords = ['patch', 'needs review']
message_count = 28.0
messages = ['194252', '195317', '195343', '196068', '196070', '196071', '196078', '196103', '196117', '196401', '196409', '196418', '196419', '196421', '196424', '196467', '196475', '204822', '223775', '223776', '223788', '223797', '223815', '223817', '223820', '229090', '229347', '229353']
nosy_count = 6.0
nosy_names = ['pitrou', 'vstinner', 'neologix', 'python-dev', 'sbt', 'pturing']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue18643'
versions = ['Python 3.5']

[neologix]

socketpair() is quite useful, notably for tests.
Currently, it's not defined on Windows.
Since it's rather easy to implement, it would be nice to have it, if not in the stdlib, at least in test.support.

[neologix]

I was thinking about dropping the C wrapper from socketmodule.c, and
replacing it with a pure Python implementation (e.g. the one posted by
Richard on python-dev).
What do you think?

[sbt]

Do you mean you want to use a pure python implementation on Unix?

Then you would have to deal with AF_UNIX (which is the default family for socketpair() currently). A pure python implementation which deals with AF_UNIX would have to temporarily create a listening socket on the file system and deal with races like tempfile.mkstemp() does. I think it is simpler to only use the pure python implementation on Windows.

BTW, it is possible for another process to connect to the address we temporily bind to. Neither my version nor the one in tulip handle that correctly.

[neologix]

Here's a patch.

Note that I'm still not sure whether it belong to the socket module or test.support.

[vstinner]

On Linux, many tests of test_socket are failing with the pure Python implementation.

[neologix]

On Linux, many tests of test_socket are failing with the pure Python
implementation.

The only failing tests I see are due to the fact that the Python
implementation uses AF_INET instead of AF_UNIX, which is normal since
Windows doesn't have AF_UNIX.
Do you see other failures?

[vstinner]

Do you see other failures?

I applied your patch and I just replace the hasattr with:
hasattr(_socket, "Xsocketpair"). test_socket failures:

======================================================================
FAIL: test_sendall_interrupted (test.test_socket.GeneralModuleTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
1259, in test_sendall_interrupted
    self.check_sendall_interrupted(False)
  File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
1248, in check_sendall_interrupted
    c.sendall(b"x" * (1024**2))
AssertionError: ZeroDivisionError not raised

======================================================================
FAIL: test_sendall_interrupted_with_timeout
(test.test_socket.GeneralModuleTests)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
1262, in test_sendall_interrupted_with_timeout
    self.check_sendall_interrupted(True)
  File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
1248, in check_sendall_interrupted
    c.sendall(b"x" * (1024**2))
AssertionError: ZeroDivisionError not raised

======================================================================
FAIL: testDefaults (test.test_socket.BasicSocketPairTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
3640, in testDefaults
    self._check_defaults(self.serv)
  File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
3630, in _check_defaults
    self.assertEqual(sock.family, socket.AF_UNIX)
AssertionError: 2 != 1

======================================================================
FAIL: testDefaults (test.test_socket.BasicSocketPairTest)
----------------------------------------------------------------------

Traceback (most recent call last):
  File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
261, in _tearDown
    raise exc
  File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
273, in clientRun
    test_func()
  File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
3637, in _testDefaults
    self._check_defaults(self.cli)
  File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
3630, in _check_defaults
    self.assertEqual(sock.family, socket.AF_UNIX)
AssertionError: 2 != 1

[neologix]

self.assertEqual(sock.family, socket.AF_UNIX)

AssertionError: 2 != 1

This is normal.

======================================================================

FAIL: test_sendall_interrupted (test.test_socket.GeneralModuleTests)
----------------------------------------------------------------------

> Traceback (most recent call last):
>   File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
> 1259, in test_sendall_interrupted
>     self.check_sendall_interrupted(False)
>   File "/home/haypo/prog/python/default/Lib/test/test_socket.py", line
> 1248, in check_sendall_interrupted
>     c.sendall(b"x" * (1024**2))
> AssertionError: ZeroDivisionError not raised

Not this one.
I can't reproduce it on my Linux box, could you post the result of:

$ strace -ttTf ./python -m test -v -m test_sendall_interrupted test_socket

[neologix]

Alright, I think I know what's happening.

The Python implementation uses a TCP socket, whereas the native
implementation uses AF_UNIX socket.
The maximum size of data that can be written to a socket without
blocking is given by its send/receive buffers.
On Linux, the default buffer sizes are set by:
net.core.(r|w)mem_default

but for TCP sockets, its set by:
net.ipv4.tcp_(r|w)mem

So on your machine, you probably have tcp_(r|w)mem quite larger than
(r|w)mem, so the sendall test doesn't write enough data to the socket
to block.

The solution is simply to increase the amount of data written.
Could you try the attached patch?

If it works I'll commit it, because the test isn't really reliable
(i.e. it could fail on a machine with a large (r|w)mem_default).

[neologix]

Victor, did you have a chance to test the patch?

[vstinner]

Victor, did you have a chance to test the patch?

I tested socketpair.diff (modified to use it even on Linux) on another computer, and test ran successfully.

On my main computer (Fedora 18, Linux kernel 3.9.4), the test is failing. With sendall_write.diff, the test does pass. test.support.PIPE_MAX_SIZE=4194305 (4 MB) on this computer. I don't know for the other one.

[neologix]

Since I'll update socket tests to also use support.PIPE_MAX_SIZE, maybe we should rename this variable while we're at it?
Anybody can think of a name that would work both for sockets and pipes?

[vstinner]

"Since I'll update socket tests to also use support.PIPE_MAX_SIZE, maybe we should rename this variable while we're at it?"

Why not using a different value (constant) for sockets?