Allowing multiple instances of IDLE with sub-processes

[taleinat]

BPO	1529142
Nosy	@rhettinger, @kbkaiser, @taleinat
Superseder	bpo-5487: Parts of Tkinter missing (but not when running from IDLE)
Files	IDLE_random_port.patch: initial patch IDLE_automatic_ports.diff: Patch to let the socket get an automatic port

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/kbkaiser'
closed_at = <Date 2009-04-04.07:35:44.342>
created_at = <Date 2006-07-26.17:45:06.000>
labels = ['expert-IDLE']
title = 'Allowing multiple instances of IDLE with sub-processes'
updated_at = <Date 2009-04-26.03:13:53.485>
user = 'https://github.com/taleinat'

bugs.python.org fields:

activity = <Date 2009-04-26.03:13:53.485>
actor = 'kbk'
assignee = 'kbk'
closed = True
closed_date = <Date 2009-04-04.07:35:44.342>
closer = 'kbk'
components = ['IDLE']
creation = <Date 2006-07-26.17:45:06.000>
creator = 'taleinat'
dependencies = []
files = ['7427', '12180']
hgrepos = []
issue_num = 1529142
keywords = ['patch']
message_count = 16.0
messages = ['50763', '50764', '65460', '65547', '65560', '65563', '76517', '76586', '76716', '80470', '80583', '80610', '83738', '85013', '85379', '86597']
nosy_count = 6.0
nosy_names = ['rhettinger', 'kbk', 'ggenellina', 'taleinat', 'gpolo', 'weeble']
pr_nums = []
priority = 'normal'
resolution = 'accepted'
stage = None
status = 'closed'
superseder = '5487'
type = None
url = 'https://bugs.python.org/issue1529142'
versions = ['Python 3.1', 'Python 2.7']

[taleinat]

(See patch 1201569 "allow running multiple instances of
IDLE" for previous discussion on this topic)

To summarize discussion up to this point: There is a
problem doing this on Windows since more than one
listening socket can be openned on a single port.

I suggest we "dodge" the Windows problem for now by
having IDLE try a random port every time (like in the
IDLEfork patch 661363) - thus collisions will be kept
to a minimum. If we choose among over 10,000 ports,
most users will never encounter a port collision. And
even when a collision does happen, it will probably be
detected and properly dealt with - collision
non-detection errors are, as Kurt mentioned, erratic.

This patch chooses a port from range(49152, 65536),
which are the 'dynamic' ports as described by the DCCP
(see http://www.iana.org/assignments/port-numbers), and
avoids known 'dangerous' ports (used by trojans, worms,
etc.) which I gathered by Googling for half an hour.

I replaced socket.timeout with select(), since I find
its timeout mechanism is more reliable. (Currently,
Python's socket's timeout mechanism raises unexpected,
unexplained errors on Windows.)

I also changed the flow a bit, so if a timeout occurs
after a sub-process is spawned, the subprocess is killed.

IMO This requires thorough testing. I tested it for
half an hour on my WinXP Python2.4.3, and after fixing
typos and such, everything seems to work.

[taleinat]

After removing the SO_REUSEADDR flag, this has been working without any pauses, delays or hitches on my WinXP Python2.5 install. I sometimes have as many as 8 separate instances of IDLE running in parallel, with no side effects.

About the SO_REUSEADDR flag, I'm wondering if this is still required on Posix systems now that we're choosing random ports. If so, we should choose whether to use this flag based on the OS.

Is there any good reason -not- to choose a random port? I can't think of any.

[taleinat]

I really, really wish we could get this in for Python2.6 - this issue is
a major drawback for beginners, for whom IDLE is mostly intended.

Perhaps not this specific patch; I am willing to work on cleaning up the
code and getting it tested by users on different platforms. Is there any
chance of this happening? What can I do to get this moving again?

[gpolo]

Works fine for me, I just dislike that Port binding error message
because it is a bit misleading. It says "IDLE can't bind any TCP/IP
port, ..." which is not entirely true.

[taleinat]

Thanks for taking the time to review this patch, Guilherme. For the
record, on which OS+Python have you tested this?

[gpolo]

I tested under Linux (Ubuntu distro), Python trunk

[weeble]

Is this ever likely to make it into IDLE? Running without a subprocess
in Windows appears to interact badly with the multiprocessing library
(attempting to spawn a process just creates a new IDLE window). I
couldn't figure out how to apply the patch on Windows - I couldn't tell
if the errors were because I was using the patch program wrong or
because the file is different now. Nevertheless, I made the changes
manually and they seemed to work. What needs to happen for this issue to
be fixed, whether by this patch or another solution?

[taleinat]

First of all, the multiprocessing module doesn't work fully in an
interactive shell.

From the Python2.6 docs:
"Note Functionality within this package requires that the __main__
method be importable by the children. This is covered in Programming
guidelines however it is worth pointing out here. This means that some
examples, such as the multiprocessing.Pool examples will not work in
the interactive interpreter."

And later on that same page (under Programming Guidelines -> Windows):
"Make sure that the main module can be safely imported by a new Python
interpreter without causing unintended side effects (such a starting a
new process)."

This is probably why IDLE is acting up for you. I would be surprised
if using the multiprocessing module will work well in IDLE either with
or without a sub-process.

As per the patch, someone should just work up an updated one with all
of the insight gained in the last several years (!) regarding this.

It should just do two things:

1. remove the use of the SO_REUSEADDR flag in the sub-process spawning
   code -- see http://bugs.python.org/issue1201569
2. pass zero as the argument for the port number, which instructs the
   underlying socket library to select an available port, and later
   retrieve the selected port from the socket object -- see
   http://mail.python.org/pipermail/idle-dev/2008-June/002687.html

The other changes in my original patch, while they do work, are
unnecessarily complex and make it much harder to get the patch
accepted. They should also not be significant if the spawning delays
are fixed.

If you'd like to work up such a patch and get it reviewed and
committed, please be my guest! I will help you in any way I can if you
take this up. Otherwise I guess it will wait until I find the time and
energy...

[weeble]

Okay, I've uploaded a patch. As suggested, it passes 0 for the port
number and the port is automatically assigned. It then extracts the port
from the socket and passes it to the subprocess. Note that this means
the subprocess inherits the listening socket, which I gather is less
than ideal. However, as far as I can tell, this happens anyway if the
subprocess is restarted, since the socket remains bound and listening
the whole time. The patch also removes the use of SO_REUSEADDR because
it is no longer needed.

This works without a problem for me on Windows XP, but I can't test it
on anything more Unixy. I removed the three retries when it fails to
bind a socket - it's not clear when this might fail and yet still be a
good idea to retry, but I can put that back if that's something that
shouldn't be messed with.

[weeble]

I installed Ubuntu on a laptop at the beginning of January and have
applied the patch I submitted above to IDLE. I've been using it
regularly for the last few weeks and have had no problems with it.

[weeble]

A thought occurs to me: would this patch make it harder to cope with
awkward firewalls that block the connection? Are they more or less
likely to intervene when passing a port of 0 and letting it pick a port
automatically? And if they do intervene, would users be able to create
an exception if there is no small list of ports that might be used?

Apart from that, what do I need to do to move this forward? Go hunt for
more testers willing to try out the patch? Find an expert to review it?

[ggenellina]

If this patch were accepted, bpo-5065 would be a non-issue then.

[rhettinger]

Is there anyone following this thread that thinks the weeble's patch is
not ready to commit?