Build only: TestProvisioner has become very flaky

[nedtwigg]

• PR feat(FeatureClassLoader): support class loading of Java 9+ #426 passed CI, but master failed after it was merged

Failure

:testlib:test
com.diffplug.spotless.scala.ScalaFmtStepTest > equality FAILED
    java.lang.AssertionError at ScalaFmtStepTest.java:83
        Caused by: org.gradle.api.artifacts.ResolveException at ScalaFmtStepTest.java:83
            Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException at ScalaFmtStepTest.java:83
                Caused by: org.gradle.internal.resolve.ModuleVersionResolveException at ScalaFmtStepTest.java:83
                    Caused by: org.gradle.internal.resolve.ModuleVersionResolveException at ScalaFmtStepTest.java:83
                        Caused by: org.gradle.api.resources.ResourceException at ScalaFmtStepTest.java:83
                            Caused by: org.gradle.internal.resource.transport.http.HttpRequestException at ScalaFmtStepTest.java:83
                                Caused by: org.gradle.internal.impldep.org.apache.http.conn.ConnectTimeoutException at ScalaFmtStepTest.java:83
                                    Caused by: java.net.SocketTimeoutException at ScalaFmtStepTest.java:83
com.diffplug.spotless.markdown.FreshMarkStepTest > equality FAILED
    java.lang.AssertionError at FreshMarkStepTest.java:62
        Caused by: org.gradle.api.artifacts.ResolveException at FreshMarkStepTest.java:62
            Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException at FreshMarkStepTest.java:62
                Caused by: org.gradle.internal.resolve.ModuleVersionResolveException at FreshMarkStepTest.java:62
                    Caused by: org.gradle.internal.resolve.ModuleVersionResolveException at FreshMarkStepTest.java:62
com.diffplug.spotless.markdown.FreshMarkStepTest > behavior FAILED
    org.gradle.api.artifacts.ResolveException at FreshMarkStepTest.java:35
        Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException at FreshMarkStepTest.java:35
            Caused by: org.gradle.internal.resolve.ModuleVersionResolveException at FreshMarkStepTest.java:35
                Caused by: org.gradle.internal.resolve.ModuleVersionResolveException at FreshMarkStepTest.java:35
78 tests completed, 3 failed
:testlib:test FAILED
FAILURE: Build failed with an exception.

• PR Modernize build #428 also passed CI, but master continued to fail after it was merged too

> Task :lib-extra:test
com.diffplug.spotless.extra.java.EclipseJdtFormatterStepTest > testSupportedVersions FAILED
    java.lang.AssertionError
        Caused by: org.gradle.api.artifacts.ResolveException
            Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException
                Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                    Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                        Caused by: org.gradle.api.resources.ResourceException
                            Caused by: org.gradle.internal.resource.transport.http.HttpErrorStatusCodeException
com.diffplug.spotless.extra.cpp.EclipseCdtFormatterStepTest > testSupportedVersions FAILED
    java.lang.AssertionError
        Caused by: org.gradle.api.artifacts.ResolveException
            Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException
                Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                    Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                        Caused by: org.gradle.api.resources.ResourceException
                            Caused by: org.gradle.internal.resource.transport.http.HttpErrorStatusCodeException
com.diffplug.spotless.extra.wtp.EclipseWtpFormatterStepTest > testSupportedVersions[CSS] FAILED
    java.lang.AssertionError
        Caused by: org.gradle.api.artifacts.ResolveException
            Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException
                Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                    Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                        Caused by: org.gradle.api.resources.ResourceException
                            Caused by: org.gradle.internal.resource.transport.http.HttpErrorStatusCodeException
com.diffplug.spotless.extra.wtp.EclipseWtpFormatterStepTest > testSupportedVersions[HTML] FAILED
    java.lang.AssertionError
        Caused by: org.gradle.api.artifacts.ResolveException
            Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException
                Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                    Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                        Caused by: org.gradle.api.resources.ResourceException
                            Caused by: org.gradle.internal.resource.transport.http.HttpErrorStatusCodeException
com.diffplug.spotless.extra.wtp.EclipseWtpFormatterStepTest > testSupportedVersions[JS] FAILED
    java.lang.AssertionError
        Caused by: org.gradle.api.artifacts.ResolveException
            Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException
                Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                    Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                        Caused by: org.gradle.api.resources.ResourceException
                            Caused by: org.gradle.internal.resource.transport.http.HttpErrorStatusCodeException
com.diffplug.spotless.extra.groovy.GrEclipseFormatterStepTest > testSupportedVersions FAILED
    java.lang.AssertionError
        Caused by: org.gradle.api.artifacts.ResolveException
            Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException
                Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                    Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                        Caused by: org.gradle.api.resources.ResourceException
                            Caused by: org.gradle.internal.resource.transport.http.HttpErrorStatusCodeException

Since the PR's passed before they failed, it seems to be a flaky sort of failure - the very best kind!!

On my box, I get a different failure:

> Task :testlib:test
com.diffplug.spotless.kotlin.KtLintStepTest > worksShyiko FAILED
    org.gradle.api.artifacts.ResolveException at KtLintStepTest.java:47
        Caused by: org.gradle.api.internal.artifacts.ivyservice.DefaultLenientConfiguration$ArtifactResolveException at KtLintStepTest.java:47
            Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                    Caused by: org.gradle.api.internal.artifacts.ivyservice.ivyresolve.parser.MetaDataParseException
                        Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                            Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                                Caused by: org.gradle.api.internal.artifacts.ivyservice.ivyresolve.parser.MetaDataParseException
                                    Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                                        Caused by: org.gradle.internal.resolve.ModuleVersionResolveException
                                            Caused by: org.gradle.internal.resource.local.FileStoreException
                                                Caused by: org.gradle.api.UncheckedIOException
                                                    Caused by: java.nio.file.NoSuchFileException

78 tests completed, 1 failed

> Task :testlib:test FAILED

FAILURE: Build failed with an exception.

After I removed the travis cache, the build passed (or at least revealed an easy-to-fix bug in #426). I think it must be that TestProvisioner is somehow gunking up gradle's artifact resolution. On my machine, a git clean -d -X -f did not fix the problem. Neither did nuking ~/.gradle/caches...

We've got a man on the inside now ;-) . Hey @JLLeitschuh, we want to turn a list of maven coordinates into files on our harddrive inside of test code. Do you know of a better gradle API to accomplish this?

spotless/testlib/src/main/java/com/diffplug/spotless/TestProvisioner.java

Lines 53 to 68 in babc574

	Project project = ProjectBuilder.builder().build();
	repoConfig.accept(project.getRepositories());
	return (withTransitives, mavenCoords) -> {
	Dependency[] deps = mavenCoords.stream()
	.map(project.getDependencies()::create)
	.toArray(Dependency[]::new);
	Configuration config = project.getConfigurations().detachedConfiguration(deps);
	config.setTransitive(withTransitives);
	config.setDescription(mavenCoords.toString());
	try {
	return config.resolve();
	} catch (ResolveException e) {
	/* Provide Maven coordinates in exception message instead of static string 'detachedConfiguration' */
	throw new ResolveException(config.getDescription(), e);
	}
	};

[nedtwigg]

Since updating the gradle version, this has become a frequent problem. We should disable travis caching until we can resolve this.

[nedtwigg]

Build caching is turned back on. I'm not super confident that it is really fixed, but my box had problems, and those problems are now fixed, so I'm closing this for now.

[nedtwigg]

It's now failing close to 100% of the time, but only for KtLintStepTest. Interestingly, that is the only test that uses jcenter(), all of the rest use mavenCentral(). I removed this for now in #499. There is a gradle bug for resolving this here: gradle/gradle#11752