Increase Py_AddPendingCall array size

[felipecruz]

BPO	16565
Nosy	@jcea, @amauryfa, @pitrou, @felipecruz

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2012-11-27.20:37:34.406>
labels = ['3.7', 'expert-IO']
title = 'Increase Py_AddPendingCall array size'
updated_at = <Date 2017-09-04.23:43:42.123>
user = 'https://github.com/felipecruz'

bugs.python.org fields:

activity = <Date 2017-09-04.23:43:42.123>
actor = 'gregory.p.smith'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['IO']
creation = <Date 2012-11-27.20:37:34.406>
creator = 'felipecruz'
dependencies = []
files = []
hgrepos = []
issue_num = 16565
keywords = []
message_count = 9.0
messages = ['176491', '176568', '176569', '176613', '176633', '176644', '176649', '176651', '176659']
nosy_count = 4.0
nosy_names = ['jcea', 'amaury.forgeotdarc', 'pitrou', 'felipecruz']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = None
url = 'https://bugs.python.org/issue16565'
versions = ['Python 3.7']

[felipecruz]

Current pending calls limit is too small and it can be easily reached in very intensive async file io applications.

There is a little hack in pyaio[1] which sleeps if Py_AddPendingCall returns < 0 but It's not totally clear to me why the size of pendind calls array is only 32.

[1] https://github.com/felipecruz/pyaio

[amauryfa]

This array would still have a fixed size.
Could you arrange your code so that you enqueue pending calls elsewhere, and call Py_AddPendingCall only once until the pending items have been processed?

[pitrou]

I just took a look at felipecruz/pyaio@127372b and it looks a bit silly to me: if the completion handler is called in a separate thread, then the completion handler can just take the GIL and put the results in a regular Python queue.

In short, using signals for any kind of high-speed I/O sounds completely archaic to me.

[felipecruz]

Just confirmed that signals is not a viable option. Is too slow, as Antonie already pointed. It's almost 5 times slower than with SIGEV_THREAD.

The problem now is:

If I use Py_AddPendingCall, the tests can hang sometimes. I can still follow Amaury suggestion tough.

If I try to acquire the GIL PyGILState_Ensure() call PyObject_CallObject and release the GIL in the aio_completion_handler function a SEGFAULT occurs in 2.7.3 but not in 3.3 and newer..

This branch works only on 3.3 and newer: https://github.com/felipecruz/pyaio/tree/feature/check_no_pending_call

This other branch will segfault 2.7.3 with just acquire and release GIL on the completion handler:
https://github.com/felipecruz/pyaio/tree/feature/py27_gil_error

Since this looks very strange, can someone confirm this behavior?

Tested on a Ubuntu12 64.

[pitrou]

If I try to acquire the GIL PyGILState_Ensure() call
PyObject_CallObject and release the GIL in the aio_completion_handler
function a SEGFAULT occurs in 2.7.3 but not in 3.3 and newer..

Well, can you post a traceback?

[felipecruz]

Running test_aio_read
[New Thread 0x7ffff7ff7700 (LWP 20681)]
[New Thread 0x7ffff61ff700 (LWP 20682)]

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff61ff700 (LWP 20682)]
sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
34 ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S: No such file or directory.
(gdb) backtrace
#0 sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
#1 0x000000000051b053 in PyThread_release_lock (lock=0x0) at Python/thread_pthread.h:346
#2 0x00000000004c868f in PyEval_ReleaseLock () at Python/ceval.c:265
#3 0x0000000000501974 in PyThreadState_DeleteCurrent () at Python/pystate.c:321
#4 0x0000000000502116 in PyGILState_Release (oldstate=PyGILState_UNLOCKED) at Python/pystate.c:652
#5 0x00007ffff660ed44 in aio_completion_handler (sigval=...) at pyaio/core.c:26
#6 0x00007ffff6409cdc in notify_func_wrapper (arg=0x7ffff00008c0) at ../sysdeps/pthread/aio_notify.c:45
#7 0x00007ffff7bc4e9a in start_thread (arg=0x7ffff61ff700) at pthread_create.c:308
#8 0x00007ffff71f14bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#9 0x0000000000000000 in ?? ()

[pitrou]

Is it with a debug Python build? PyThread_release_lock (lock=0x0) is a bit worrying.

[felipecruz]

Yes! 2.7.3 build with pydebug.

[felipecruz]

Without debug backtrace

#0 sem_post () at ../nptl/sysdeps/unix/sysv/linux/x86_64/sem_post.S:34
#1 0x00000000004d456e in PyGILState_Release ()
#2 0x00007ffff59b9cdc in notify_func_wrapper (arg=0x7ffff00008c0) at ../sysdeps/pthread/aio_notify.c:45
#3 0x00007ffff7bc4e9a in start_thread (arg=0x7ffff59b5700) at pthread_create.c:308
#4 0x00007ffff69b64bd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112
#5 0x0000000000000000 in ?? ()

[kumaraditya303]

Closing because of lack of activity for a decade.