Coveralls.io

[singingwolfboy]

Measure test coverage, report results to Coveralls, display a badge in the README. Note that this depends on the tests measuring code coverage -- there's a pull request for that here: pyinvoke/invocations#5

[bitprophet]

Thanks, have been meaning to poke more at coverage and linting. Will see if I can get to this sometime soon.

[singingwolfboy]

@bitprophet ping

[justinabrahms]

I'd (humbly) offer checking out codecov.io. It will report on how a particular PR is covered, in addition to what coveralls does, which is report on how the PR affects coverage.

I'm much less interested in (+0.02% coverage) than I am (This PR is 48% covered).

[bitprophet]

a) I've been poking more at vanilla coverage.py lately and it is real nice, so this topic is on my mind (also commented on the invocations PR, since I accidentally wrote my own coverage method)
b) I agree with @justinabrahms that coveralls' output (as seen over on paramiko/paramiko for the last, like, yearish) is really not very helpful, I should check out codecov.io sometime.

[bitprophet]

Double checked the current state of features for coveralls.io vs codecov and from their websites at least, codecov does still look more useful, coveralls continues to have the "+N% coverage" focus which is less useful.

Will test out actually using codecov and if I like how it looks, will add that to travis.

[bitprophet]

Hrm, don't see any way to tell codecov to give flags to the report generation; currently my coverage stuff has to tell coverage html to --include invoke/* and --omit invoke/vendor/*, otherwise it thinks my suite only covers like 15% because it doesn't test most of the vendored material.

Similarly I don't see a way to give this info to coveralls.io.

Digging further to see why I opted to do the filtering at html-generation time, maybe I'll get lucky and there's a way to do it at coverage-generation time instead. Otherwise this will be kind of a no-op and I'll just have to jury-rig something stupid like "generate coverage report, then inspect it for the coverage percentage".

[bitprophet]

Sweet, apparently at the time I was ignorant of the use of .coveragerc or it didn't exist - my suite needs to run under a nosetests wrapper, so coverage run isn't a good option. But nosetests --with-coverage honors .coveragerc so I think I am in business.

[bitprophet]

Test upload seems to work reasonably well: https://codecov.io/github/pyinvoke/invoke?ref=07a956e789c25b721bd08442624b60c70efd47a8

It's like 0.5% less coverage than native coverage html reports, and this seems to be because it's only seeing about 1/2 of the branches - coverage html shows 584 branches but codecov only shows 289.

The 1/2-ness is constant across all covered files, which is...pretty bizarre. Can't find much online about this besides a probably-not-related twitter convo: https://twitter.com/glenathan/status/603883608651264000

That said, when it represents such a small amount-of-coverage for now, and only seems to affect branch coverage which is itself always a dicey proposition, Good Enough For Now.

[justinabrahms]

Glad to hear that. The founder there is pretty responsive, in my
experience. Consider messaging him.

On Monday, December 21, 2015, Jeff Forcier [email protected] wrote:

Test upload seems to work reasonably well:
https://codecov.io/github/pyinvoke/invoke?ref=07a956e789c25b721bd08442624b60c70efd47a8

It's like 0.5% less coverage than native coverage html reports, and this
seems to be because it's only seeing about 1/2 of the branches - coverage
html shows 584 branches but codecov only shows 289.

The 1/2-ness is constant across all covered files, which is...pretty
bizarre. Can't find much online about this besides a probably-not-related
twitter convo: https://twitter.com/glenathan/status/603883608651264000

That said, when it represents such a small amount-of-coverage for now, and
only seems to affect branch coverage which is itself always a dicey
proposition, Good Enough For Now.

—
Reply to this email directly or view it on GitHub
#144 (comment).

Sent via mobile phone.

[bitprophet]

Good to know, thanks again @justinabrahms!

Re: other bits in this PR, codecov's equivalent status badge can be found on https://codecov.io/github/pyinvoke/invoke/features/badges so I'll work that in, and then preserve the Travis status badge too cuz why not :) EDIT: oh right, that stuff's in our website sidebar, so having it in the README is kind of meh. May take a quick stab at adding it to the sidebar under the Travis hook.

[bitprophet]

Welp Alabaster 0.7.7 is now out with Codecov button support :3

[bitprophet]

Set CodeCov PR adjusted-line coverage to 90% for now, partly cuz that'd be nice, partly because I assume lots of PRs will fail and I want to see what failure looks like initially :3