WIP: Json documentation generation

[daspecster]

scripts/generate_json_docs.py is now generating parsable JSON for the gcloud-common site generator.
As well as building a types.json file.

Several things are still in progress...

• Extracting code examples is not working consistently.
• The table of contents needs to be updated. (just switched to listing all classes)
• Class linking should be added. i.e. class:gcloud.datastore._generated.query_pb2.Query
• Source Code links are broken for base class docs
• Not yet sure how to handle __init__.py's
• "Getting Started" and "Guide" type pages need to be ported.(done for what we have but should add more).
• Running generate_json_docs.py needs to be added to the build process and documented better.
• Find all directives that are not filtered and filter/handle them (for now anyway).
• Possibly add auto-href generation for link in text for easier navigation out site docs.
• Figure out how to handle snippets. This should probably be in a different PR after this launches
• Bring in Code snippets Add ability to pull snippets when generating JSON documentation #1951
• Trim back TOC.
• Switch to live gh-pages committing

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request) and all commit authors, but as best as we can tell these commits were authored by someone else. If that's the case, please add them to this pull request and have them confirm that they're okay with these commits being contributed to Google. If we're mistaken and you did author these commits, just reply here to confirm.

[dhermes]

@daspecster From https://api.github.com/repos/GoogleCloudPlatform/gcloud-python/pulls/1775/commits it appears some of the commits have @jgeewax and @callmehiphop emails in them (which is why the CLA bot is angry).

[dhermes]

High level comment: It looks like you've had to write the majority of the parser from scratch (i.e. have your own representations of Module / Class / etc.) The pdoc.import_module behavior seems to just be a fancy AST parser?

[daspecster]

Yeah, I think I'm probably going to drop pdoc and just use inspect.

[daspecster]

Also, @dhermes for the ci bot, should I just squash those commits? I didn't want to leave out the log of their contributions.

[dhermes]

I don't particularly mind about the CI, though those initial commits aren't necessarily needed for your work.

[jgeewax]

Nice -- this is awesome. Thanks for cranking on it.

[googlebot]

CLAs look good, thanks!

[googlebot]

We found a Contributor License Agreement for you (the sender of this pull request) and all commit authors, but as best as we can tell these commits were authored by someone else. If that's the case, please add them to this pull request and have them confirm that they're okay with these commits being contributed to Google. If we're mistaken and you did author these commits, just reply here to confirm.

[tseaver]

The "continuation line" lint failures are obnoxious: finding the style which passes both pep8 and pylint is always a bit of trial and error for me. The others are fairly straightforward fixes.

[daspecster]

Ok! Have another look through and LMKWYT

http://daspecster.github.io/gcloud-python/

[daspecster]

Sorry, forgot to tag @tseaver and @dhermes on that last comment.

[tseaver]

The docstring examples on the datastore landing page still look borked:

http://daspecster.github.io/gcloud-python/#/docs/master/gcloud/datastore

The snippet captions appear to have regressed:

http://daspecster.github.io/gcloud-python/#/docs/master/gcloud/bigquery

http://daspecster.github.io/gcloud-python/#/docs/master/gcloud/pubsub

[daspecster]

I think I somehow squashed the snippet title code out. I'll try and redo it real quick.

[daspecster]

@tseaver fixed! http://daspecster.github.io/gcloud-python/#/docs/master/gcloud/pubsub
I fixed the .. doctest:: thing with #2006.

Not sure how that got lost, I think I may have lost the commit in rebasing or something.

[tseaver]

@daspecster The http://daspecster.github.io/gcloud-python/stable/bigquery-usage.html page is still showing the pre-snippet examples: maybe that will get fixed when you merge?

[daspecster]

@tseaver Oh sorry, I pushed gh-pages when I was doing #2009 and #2008.
It's back now though. http://daspecster.github.io/gcloud-python/#/docs/master/gcloud/pubsub

[daspecster]

Let me know if there is anything that's a show stopper here. I would like to try and get this merged today if possible.

[tseaver]

LGTM

[daspecster]

I'm going to merge this at 2pm EST unless I hear otherwise. Just to give everyone a chance to chime in.

[daspecster]

Travis master build that I'm watching for this https://travis-ci.org/GoogleCloudPlatform/gcloud-python/builds/146717128