Emit major version based canonical URLs for docs

[ncoghlan]

BPO	26355
Nosy	@birkenfeld, @ncoghlan, @ezio-melotti, @berkerpeksag, @Carreau, @wheerd
PRs	[Do Not Merge] Convert Misc/NEWS so that it is managed by towncrier #552
Files	cannonical-doc-for-3.4plus.patch cannonical-doc-for-2.7.patch cannonical-doc-for-2.6.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ncoghlan'
closed_at = <Date 2017-02-09.16:07:27.661>
created_at = <Date 2016-02-13.08:52:19.337>
labels = ['type-feature', '3.7', 'docs']
title = 'Emit major version based canonical URLs for docs'
updated_at = <Date 2017-03-31.16:36:30.069>
user = 'https://github.com/ncoghlan'

bugs.python.org fields:

activity = <Date 2017-03-31.16:36:30.069>
actor = 'dstufft'
assignee = 'ncoghlan'
closed = True
closed_date = <Date 2017-02-09.16:07:27.661>
closer = 'ncoghlan'
components = ['Documentation']
creation = <Date 2016-02-13.08:52:19.337>
creator = 'ncoghlan'
dependencies = []
files = ['46563', '46564', '46565']
hgrepos = []
issue_num = 26355
keywords = ['patch']
message_count = 22.0
messages = ['260227', '260237', '286319', '286336', '286339', '286342', '286344', '287139', '287191', '287203', '287226', '287249', '287419', '287425', '287430', '287431', '287432', '287433', '287437', '287441', '287445', '287448']
nosy_count = 8.0
nosy_names = ['georg.brandl', 'ncoghlan', 'ezio.melotti', 'docs@python', 'python-dev', 'berker.peksag', 'mbussonn', 'Wheerd']
pr_nums = ['552']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue26355'
versions = ['Python 2.7', 'Python 3.5', 'Python 3.6', 'Python 3.7']

[ncoghlan]

Based on a recent comment on one of the mailing lists, I spent a bit of time looking into canonical URLs and their implications for how search engines treat versioned documentation.

The most relevant resource for that in relation to the CPython docs appears to be this page on ReadTheDocs: http://docs.readthedocs.org/en/latest/canonical.html

I sometimes encounter direct links to particular 3.x versions in search results, so I'm wondering if it might be desirable to set up the Python 3 docs to report their canonical URL as "/3/<path>", and the Python 2 docs as "/2/<path>".

[birkenfeld]

So we'd have to add a <link> tag? Should be easy.

[ncoghlan]

Belatedly following up on this, yeah, the RTFD page indicates that the header link should look like:

<link rel="canonical" href="http://docs.python.org/3/<path>">

[Carreau]

Does this have to be implemented on the doc build of EOL pythons versions (like 2.6), or can it be a script which is ran once on these old docs ?

One of the issues I had trying to implement that on other projects was that you don't know in advance what the future pages will be or if it will be gone. So you can go this route[1], in which case it's a 10 line fix that I'm happy to contribute also here.

Typically https://docs.python.org/2/c-api/string.html has no cannonical on 3 whouch you can't know while building with sphinx.

Or I though about injecting an html comment with a html comment that you search and replace once you publish the "new" version and only if the target page exists on the new version.

So :

• is that ok to have non existing cannonical ?
• is that ok if adding <rel>s on docs is (or requires) a post-sphinx script ?

1: https://github.com/xonsh/xonsh/pull/1914/files

Thanks

[Carreau]

Here is a patch that add a canonical link to the current documentation that should apply cleanly to 3.4 and above. I can do similar for older versions.

[wheerd]

I create the last diff without creating a commit, so maybe this one works better.

[wheerd]

Sorry, I accidentally replied to the worng issue -.-

[ncoghlan]

Thanks Matthias!

Regarding 2v3, the layout differences aren't a problem, since the canonical URLs are separate (/2/* vs /3/). That's one of the benefits I actually hope for with this change - due to PEP-430, deep links still go to the Python 2 documentation by default, and once this change is made in the Python 2.7 branch it should teach search engines that those should start being presented in results as "/2/" qualified links.

We also don't tend to make wholesale changes to the URL layouts in the docs in X.Y releases, so I think the assumption of "the relative path of this page won't change" is fine.

Georg, Berker - any further thoughts before we make this change to 3.4+ and the 2.7 docs?

[Carreau]

Thanks Nick,

I can work on the similar patch for Python 2.7.
I'll wait for the migration to GitHub which IIRC should be soon.

Side question, is there some metrics (like google analytics) to know how much traffic there is on older Python docs and if it's worth patching ?

[berkerpeksag]

Patch looks good to me too, thanks! Just left a super minor comment on Rietveld.

You might want to commit this to 3.5+ because we don't daily build 3.4 docs anymore: https://github.com/python/docsbuild-scripts/blob/master/build_docs.py#L30

I didn't see any links from 2.6 or older Python 2 versions in search results before so I don't have a strong opinion on backporting the patch to 2.7 (I probably wouldn't bother with Python 2 docs anymore :))

[ncoghlan]

OK, I've marked 2.7 and 3.5+ as the minimal set of versions to get the change via the CPython source repo.

However, looking at the results of https://www.google.com/search?q=python+httplib and https://www.google.com/search?q=python+http+client I think it's going to be worth backfilling the old branches as well - having multiple different versions of the 3.x documentation showing up in search results isn't helpful to anyone.