Move Windows external libs from <src>\..\ to <src>\externals

[zware]

BPO	17896
Nosy	@loewis, @terryjreedy, @db3l, @tjguk, @tpn, @zware, @zooba
Files	issue17896-default.diff issue17896-3.4.diff issue17896-2.7.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/zware'
closed_at = <Date 2014-11-02.03:57:12.415>
created_at = <Date 2013-05-03.17:18:26.396>
labels = ['extension-modules', 'type-feature', 'OS-windows', 'build']
title = 'Move Windows external libs from <src>\\..\\ to <src>\\externals'
updated_at = <Date 2015-02-09.07:05:13.835>
user = 'https://github.com/zware'

bugs.python.org fields:

activity = <Date 2015-02-09.07:05:13.835>
actor = 'python-dev'
assignee = 'zach.ware'
closed = True
closed_date = <Date 2014-11-02.03:57:12.415>
closer = 'zach.ware'
components = ['Build', 'Extension Modules', 'Windows']
creation = <Date 2013-05-03.17:18:26.396>
creator = 'zach.ware'
dependencies = []
files = ['37058', '37059', '37060']
hgrepos = []
issue_num = 17896
keywords = ['patch']
message_count = 20.0
messages = ['188309', '223484', '223496', '225812', '230088', '230092', '230102', '230108', '230109', '230110', '230122', '230199', '230252', '230477', '230479', '230480', '230518', '230574', '235552', '235595']
nosy_count = 10.0
nosy_names = ['loewis', 'terry.reedy', 'db3l', 'tim.golden', 'trent', 'jeremy.kloth', 'BreamoreBoy', 'python-dev', 'zach.ware', 'steve.dower']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue17896'
versions = ['Python 2.7', 'Python 3.4', 'Python 3.5']

[zware]

PCbuild\readme.txt has a comment from Trent Nelson dated 2-April-2008 suggesting moving the external library location for Windows from "<cpython checkout>\..\" to "<cpython checkout>\external\" to make switching between versions easier. I've gotten rather annoyed with having all of the external libs cluttering the folder holding my cpython checkout, so I've written a patch that moves them :).

The patch actually moves the external libs into "<checkout>\external-34", because in most cases it's best to have separately built externals for each Python version. The idea is to then have an "external-35" after 3.4 is released, and possibly "external-27" and "external-33" if it is permissible to backport this kind of change. There are only four places that would need to be changed in other versions; PCbuild/pyproject.props:19, PCbuild/rt.bat:41, Tools/buildbot/external-common.bat:4 and 5, and Lib/tkinter/_fix.py:52. All four are simple replacements which I believe could be automated by the release script. PCbuild/build_ssl.py is patched to find the externals dir in PCbuild/pyproject.props while finding the OpenSSL version there.

A few of the benefits to this that have occurred to me are:

• the fact that the dir holding your cpython checkout doesn't get cluttered up with external libs
• it becomes very easy to rebuild all of the externals (just rmdir /s /q external-34 and run external(-amd64).bat again)
• no possibility of problems arising from one version trying to use an external lib built by/for another version

The only real downsides I'm aware of are that it is a change from what everyone is accustomed to, and that it will use up a bit more disk space (which could be mitigated by using junctions, as Trent suggested in the comment in readme.txt). Also, committing this would cause an extra long build time on the buildbots due to having to recompile Tcl/Tk and OpenSSL.

Everything builds and works for me using the buildbot scripts after the patch. All comments welcome :)

Thanks,

Zach

[BreamoreBoy]

I also find the location annoying and would be happy to see it moved.

Regarding the "extra long build time on the buildbots due to having to recompile Tcl/Tk and OpenSSL" is this a one off the first time you run after the commit, or does it always happen because the buildbots run some form of clean? Sorry if this sounds like a daft question but I haven't the faintest idea how the various buildbots are configured.

[zware]

After the most recent changes to the buildbot scripts, it would be every time without a patch to the clean script, but at the time I only meant a single time (which really isn't a big deal).

The patch is well out of date, I can update it if we can come to a consensus that this kind of change is a good idea.

[BreamoreBoy]

I see that this has also been raised on bpo-22262. As nobody has objected can't we just get on with it?

[zooba]

Not so keen on having separate folders for Python version, mostly because I want to avoid having the current version in too many locations. I've settled on patchlevel.h as the canonical source of the version number for my VC14 branch - everything else should read it from there.

If the patch is updated to read it from patchlevel.h then I'm +1. Otherwise +0, and I'll probably update it later myself :)

[zware]

I'm ambivalent on per-version externals dirs by now; I've since found it much easier to maintain hg-shared repos per branch. I'll go ahead with this shortly on all three branches, unless there are objections to it on 2.7 and 3.4.

[zooba]

Sounds good to me. I thought about the shared repo approach, but I don't understand fully how merging between 2.7, 3.4 and default works in that scenario (simply because I've never tried it - maybe it's really easy?)

[zware]

Basically, after:

hg clone h.p.o/cpython default
hg share default 3.4
hg share default 2.7
hg -R 3.4 update 3.4
hg -R 2.7 update 2.7

the 2.7, 3.4, and default directories are separate working copies created from the same history, each at a different revision. As soon as a changeset is created in one, it's available in the others (but the other working copies don't change). So after commiting to 2.7 (from within the 2.7 dir), cd ..\3.4 && hg graft 2.7 will try to graft your new changeset to 3.4, then cd ..\default && hg merge 3.4 will merge it with default. hg push will push the same thing from any of the three directories.

[terryjreedy]

Since all other dependency directories are dependency version specific, the only problem with a common external directory is the unversioned tcltk/. What do you plan to do with that?

[zware]

There's no change from the status quo on that front: the only change
is that the $(externalsDir) VS variable becomes "..\externals" instead
of "..\..". Is there an open issue for versioning the tcltk[64] dirs?

[terryjreedy]

I could not find an issue specifically for the tcltk problem. I just explained the problem as I know it in msg230120 of bpo-17896. Some sort of fix is required before we can merge multiple tcltk directories in isolated build directories of the sort you diagrammed in msg214138.