Vendorize dependencies

[bitprophet]

(Was: Deal with Fluidity dependency)

This desc is out of date, see this comment below for new game plan.

Original desc follows.

---

We're using Fluidity for state machine fun, but their packaging is FUBARed right now: version 0.2.1 has a useful-for-debugging hook not in 0.2.0, but 0.2.1 is not actually tagged or on PyPI so it's not possible (AFAIK) to obtain it via a regular pip install.

I'll ping the devs there but if they are unable to comply, options are:

• Vendorize it. Meh. Always annoying re: syncing with upstream. OTOH it's not a big or complex package nor is it heavily developed, so.
• Publish our own copy of it, either via Github tag-url or another name on PyPI. Also meh, but useful if we ever needed to fork it for real sometime (unlikely.)

[bitprophet]

nsi-iff/fluidity#4

(Somehow, the false ticket on Lexicon, this one, and the above -- all number 4!)

[bitprophet]

Posted some thoughts on the matter. I doubt I'll get any seriously worldview-changing feedback, but do want to see what comes up.

Truly torn on what I think is the right choice for us re: Fluidity. Leaning towards vendorizing because of the "can work standalone" angle for this type of tool; though I will probably look at each new dependency independently. Anything with high turnover (e.g. Clint) will almost definitely work best non-vendorized.

[bitprophet]

Yea. Going to try vendorizing things for now (maybe using subtree merging to help with the pain of keeping up with external modules) with a strong possibility of undoing that (and using e.g. Github tarball URLs, which it turns out setuptools can use, pip isn't absolutely required) before 1.0.

[bitprophet]

I don't know why this is closed, I don't think I actually pulled down/vendorized Fluidity yet (it's not in the source tree.) I do still think vendorizing it is the right approach though.

[bitprophet]

Tried starting out w/ the git subtree merge approach (good explanation here), where you basically:

• Add a remote to the dependency's repo, and a branch based on that repo's e.g. master (which is a "branch" in name only as it looks like the dep, not your project);
• Use e.g. git read-tree --prefix vendor/fluidity -u fluidity_branch to write out a branch from that remote/repo into a subdirectory of your actual project;
• Commit that initial read/dump to start;
• To obtain updates, git merge -s subtree --squash --no-commit fluidity_branch, which is similar to the initial read-tree in that you get one big rolled-up commit representing multiple upstream commits.

Unfortunately: this approach results in the dep's top level source directory inside your e.g. vendor/fluidity/ folder. Which is not a valid Python subpackage (that would be e.g. vendor/fluidity/fluidity) and thus can't be imported via e.g. import invoke.vendor.fluidity.

Poking for solutions now, but given that this approach essentially skips using Git's history mechanisms anyway, I don't see what we really gain over just:

• Stick a checkout of fluidity in my projects folder;
• Manually copy its fluidity Python package folder into invoke/vendor/fluidity, noting somewhere which commit it was;
• Commit that.
• To get updates, cd my_copy_of_fluidity; git diff $PREV_COMMIT > lol.patch; cd invoke/vendor; patch lol.patch or similar (and then update $PREV_COMMIT).

Git submodules would take care of the $PREV_COMMIT tracking business, but requires downstream devs or anybody trying to do pip install -e https://github.com/my/project#blahblah to take extra steps, or maybe even be hosed. Also may still not allow selecting a specific subdir of the target repo, I forget.

Again, this is under active investigation.

[bitprophet]

To do:

• Whip up some tasks.py tasks to automate updating the vendored stuff (a la rails Rakefiles...heh)
• Pull down some initial vendorizations of Fluidity and Pexpect
• Ensure we are doing the right thing re: those projects' licenses: is simply preserving their LICENSE files within their source trees sufficient, do we need to pull them into our root folder, explicitly mention them in our README, ???
  • E.g. Requests appears to have licensing info at the top of every file within his packages/* vendorized directories, but that appears to be just how the upstream packages are normally.
  • For packages that (like I like to do) just have one LICENSE file in their source distribution, that means we have to make copying that license part of our vendorization process.
    • Fluidity is one of these
    • Pexpect has the license embedded in its header (and is also effectively a single module distribution) in addition to a standalone LICENSE file. Sounds like easiest is probably just to stick pexpect.py in vendor/ by itself.

[bitprophet]

I have performed the initial vendorizing here and it works OK for new installs. The "update/redo my vendored deps" task is still just a skeleton of comments, but that can be fleshed out whenever I actually need to upgrade something.