Queue in python 3.5 asserts/hangs with low queue size

[obmarg]

I've been attempting to update to python 3.5 today, but it has been causing some of our unit tests to fail with an assert error.

I've reproduced in the repl with this code:

import asyncio

@asyncio.coroutine
def consumer(queue, num_expected):
    for _ in range(num_expected):
        yield from queue.get()

@asyncio.coroutine
def producer(queue, num_items):
    for i in range(num_items):
        yield from queue.put(i)


queue_size = 1
producer_num_items = 5
q = asyncio.Queue(queue_size)

results = asyncio.get_event_loop().run_until_complete(
    asyncio.gather(producer(q, producer_num_items),
                   consumer(q, producer_num_items)),
)

This code causes AssertionError: queue non-empty, why are getters waiting? to be thrown. Strangely, if I increase queue_size to 3 or 4, this causes the code to hang. 5 seems to be the magic queue_size that makes problems go away.

[1st1]

I'll take a look at this before Monday.

[gvanrossum]

Looks like a dupe of #265 right?

[obmarg]

It's certainly very similar to #256, though in this case the assert is being thrown inside put rather than get/get_nowait. Possibly just the same issue manifesting in different ways though.

[manipopopo]

Suppose there is one item in the queue, and one pending putter waiting for a free slot.
Queue.get in the consumer awakes the pending putter, which will put an item into the queue at the next step of the event loop. Then Queue.get returns the item in the queue to the consumer. After that the consumer calls Queue.get again. Since the queue is empty, Queue.get becomes a pending getter.

In the next step of the event loop:

The awoken putter puts an item into the queue. The pending getter has no idea that the queue is not empty anymore. No one removes the getter from the list of pending tasks. When the producer calls Queue.put again, there is one pending getter and one item (from the awoken putter) in the queue, so the assertion

assert not self._queue, ('queue non-empty, why are getters waiting?')

fails.

[gvanrossum]

It's sad to see how hard it is to maintain a simple invariant in this code. :-( There are only three coroutines, and each only has one yield. Anyone up for a complete clean-room rewrite?

[1st1]

There is a pending pr I wanted to merge. Perhaps it will address some of these assert bugs. If not, I'll do my best to fix the bugs.

[manipopopo]

Thank you for all your assistance.
I will try it as well.

[gvanrossum]

The more the merrier. I have an idea for a fix as well.

[gvanrossum]

Here are tests reproducing both issues.

tests.diff.txt

[gjcarneiro]

Apologies if I broke this code. But the problem I fixed was much worse than this one. At least this error is explicit, the other bug caused queue items to get silently lost into the ether...

I will also take another look at this, I just have been busy with work (even today). I think what @obmarg writes in code should be supported.

I have a PR to refactor Queue a bit, but I already tested my PR and it doesn't fix the problem, unfortunately. OTOH, I think my refactor PR is a cleaner and simpler code on which to work on a possible bug fix for these issues.

[gvanrossum]

@gjcarneiro Please do check out my PR! #269

[gvanrossum]

Closed by 872597b