Create an end-to-end testing suite for workflows using a bundle server

[vdye]

• Probably also in the test/ subdirectory of the repo
• Should test a full end-to-end workflow: deploy to a separate server, clone/fetch/etc. with Git
• Tests are designed around scenarios, written from the user's perspective - treat services as a black box. End-to-end tests should look like a PM wrote them, vs. integration tests designed by an engineer.

[vdye]

I think the biggest hurdle here is deciding on a language & framework. Collecting a bunch of rough thoughts on the topic:

What kind of functionality do we want?

• Since we're testing a bunch of CLIs (git-bundle-server, git-bundle-web-server, git), the language we use here should have relatively simple "call program" syntax; bonus points if it can be configured to run via shell.
• For the sake of test reporting, whatever framework we use should be able to emit results in some kind of standardized format (e.g. junit).
• Built-in global "setup" and "teardown" functions would be nice for cleaning up test repositories.
• Should be configurable to run tests in parallel.

Which language?

Go

I'm actually leaning against Go, since using another language would really reinforce the "black-box" nature of end-to-end tests.

Shell

Shell tests are clearly the best choice for "ease of executing shell commands", but they'd involve a lot of boilerplate and, given our experience with Git's test suite, aren't particularly easy to work with. Leaning no on this one as well.

Typescript (Node.js)

This isn't a bad option, but personally it's not my first choice (npm is not my friend). That said, it's flexible and supports a number of fantastic testing frameworks.

Ruby

My personal top choice, mainly due to its extremely simple shell execution syntax. Like JS, there are a bunch of good test frameworks to choose from.

Test framework?

I really like Cucumber - tests are written in human-readable documents, those documents are portable across languages (should we want to change language), and most language integrations will output results in a standard format. I'm open to other ideas though!

[vdye]

Update: after doing some rough prototyping, I ended up on Typescript after all (mostly because its seems overall better supported in VSCode). It took some finagling to get Cucumber set up the way I wanted it, but at this point I'm able to run tests successfully. Hoping to have one or two scenarios implemented by the end of the week!