cmd/go: go modules ignores go.mod in semver repos not using semantic import versioning

[markbates]

When trying to import a package, in this case https://github.com/gobuffalo/pop, that has a semver tag >=2.0.0, in Pop's case it is v4.6.4 Go modules skips over versions that have go.mod files.

In this example Go Modules will always return v4.5.9, which is the highest version that does not have a go.mod. Because all versions above this have go.mod files, Go Modules refuses to pick them, resulting in strange results.

Setting a version explicitly will work, but letting Go Modules find it, always fails.

It would appear that this is the line throwing away the good releases https://github.com/golang/go/blob/master/src/cmd/go/internal/modfetch/coderepo.go#L137

A repo that shows the problem can be found here: https://github.com/gobuffalo/pop-vgo

What version of Go are you using (go version)?

go version go1.11rc1 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/markbates/Library/Caches/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/markbates/Dropbox/go"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/6m/vw2ck7mj32z5f63wpgfw5qk80000gn/T/go-build858485480=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

export GO111MODULE=on

go1.11rc1 mod init
cat go.mod
go1.11rc1 build -v .
go1.11rc1 mod tidy -v
cat go.mod | grep pop

What did you expect to see?

module pop-vgo

require (
	github.com/fsnotify/fsnotify v1.4.7 // indirect
	github.com/gobuffalo/pop v0.0.0-20180810203029-9f8bf0c11920
	github.com/golang/protobuf v1.1.0 // indirect
	github.com/hpcloud/tail v1.0.0 // indirect
	github.com/inconshreveable/mousetrap v1.0.0 // indirect
	github.com/kr/pretty v0.1.0 // indirect
	github.com/onsi/ginkgo v1.6.0 // indirect
	github.com/onsi/gomega v1.4.1 // indirect
	github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e // indirect
	github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041 // indirect
	golang.org/x/sys v0.0.0-20180815093151-14742f9018cd // indirect
	golang.org/x/text v0.3.0 // indirect
	google.golang.org/appengine v1.1.0 // indirect
	gopkg.in/fsnotify.v1 v1.4.7 // indirect
	gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
)

What did you see instead?

module github.com/gobuffalo/pop-vgo

require (
	github.com/cockroachdb/cockroach-go v0.0.0-20180212155653-59c0560478b7 // indirect
	github.com/fatih/color v1.7.0 // indirect
	github.com/fsnotify/fsnotify v1.4.7 // indirect
	github.com/go-sql-driver/mysql v1.4.0 // indirect
	github.com/gobuffalo/makr v1.1.2 // indirect
	github.com/gobuffalo/packr v1.13.2 // indirect
	github.com/gobuffalo/pop v4.5.9+incompatible
	github.com/gobuffalo/uuid v2.0.0+incompatible // indirect
	github.com/gobuffalo/validate v2.0.0+incompatible // indirect
	github.com/golang/protobuf v1.1.0 // indirect
	github.com/hpcloud/tail v1.0.0 // indirect
	github.com/inconshreveable/mousetrap v1.0.0 // indirect
	github.com/jmoiron/sqlx v0.0.0-20180614180643-0dae4fefe7c0 // indirect
	github.com/lib/pq v0.0.0-20180523175426-90697d60dd84 // indirect
	github.com/markbates/going v1.0.1 // indirect
	github.com/mattn/anko v0.0.6 // indirect
	github.com/mattn/go-colorable v0.0.9 // indirect
	github.com/mattn/go-isatty v0.0.3 // indirect
	github.com/mattn/go-sqlite3 v1.9.0 // indirect
	github.com/onsi/ginkgo v1.6.0 // indirect
	github.com/onsi/gomega v1.4.1 // indirect
	github.com/serenize/snaker v0.0.0-20171204205717-a683aaf2d516 // indirect
	golang.org/x/sys v0.0.0-20180815093151-14742f9018cd // indirect
	golang.org/x/text v0.3.0 // indirect
	google.golang.org/appengine v1.1.0 // indirect
	gopkg.in/fsnotify.v1 v1.4.7 // indirect
	gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect
	gopkg.in/yaml.v2 v2.2.1 // indirect
)

[thepudds]

@gopherbot, please add label modules

[thepudds]

Hi @markbates, I'm not 100% sure this is the root cause, but given 'pop' is >= v2, I believe you need to have the major version at the end of the module path in your module directive in your go.mod file.

Right now, it shows:

module github.com/gobuffalo/pop

https://github.com/gobuffalo/pop/blob/d38e78212cc54cd9317a67522137701a3f40af82/go.mod

This is covered a bit on the "Modules" wiki here:

https://github.com/golang/go/wiki/Modules#semantic-import-versioning

Could you try appending the major version to the end of the module path in the module directive in the go.mod file? I think that would mean appending /v4 if I've followed.

[markbates]

Yes, but as noted in this blog post, https://blog.gobuffalo.io/the-road-to-1-0-go-modules-c55b42af70de, doing so would cause breakage for lots of people. I would rather not create a release that would potentially harm users to something like that.

According to @spf13 the Semantic Import Versioning rule has been relaxed, so it would make sense that this would work.

The logic doesn't make sense. For packages that are already about v2 can't use Go Modules as it currently stands, or else Go Modules will ignore them. That is definitely not the logical workflow. I can use v4.5.9 because it doesn't have a go.mod file, but I can't use v4.6.4 because it does.

[thepudds]

Also, at the risk of getting ahead of ourselves, if that was the issue that was biting you, I would suggest re-purposing this issue into how it might have been easier to diagnose the issue (e.g., with a warning, or with -v output from 'go get', or ___).

edit: Sorry, got ahead of things with this comment. Please ignore this particular comment for now.

[thepudds]

Hi @markbates, I'm not sure specifically what @spf13 was referencing, but perhaps he was describing the work where Go versions 1.9.7+ and 1.10.3+ have been updated to know how to properly interpret a /v2 or higher that appears in an import path, for example:

https://go-review.googlesource.com/c/go/+/109340

As far as I am aware, that would still requiring having a /v4 at the end of the module directive in the go.mod file for 'pop' given the major version of the 'pop' module is v4.

If you are looking to still support Go 1.8.x with a v4.x.x module, then the only alternatives I am aware of are:

• creating a /v4 subdirectory, place a new go.mod within the subdirectory, append /v4 to the module path, and copy or perhaps move the code into the subdirectory (as described in the initial vgo blog posts, and also a bit on the Modules wiki link above), or
• waiting until 1.9.7 and 1.10.3 have been out long enough to be a reasonable choice for your users

All that said, I'm just sharing my current personal understanding... I'm happy to learn more myself, and I certainly don't want to mis-direct you, so please take with a grain of salt.

[markbates]

@thepudds All of that has been relaxed because it presents a chicken and egg problem for a lot of >2 repos out there.

The logic here is either coded incorrectly, which is what I think it is (a left over from earlier stricter releases), or the logic is fundamentally flawed, which I don't think it is. Not pulling perfectly good releases when searching for "latest" because they contain a go.mod makes zero sense. It's not what one would expect the behavior to be, nor does it encourage people to actively start using Go Modules as they'll break any >2 repo that adds them.

/cc @rsc

[thepudds]

Hi @markbates, and just to confirm, part of the current goal is to continue to support people using Go 1.8.x? I see on the Buffalo site:

Buffalo has a minimum Go dependency of 1.8.1.

I'm guessing the intent is to continue to support 1.8.x right now, but I thought worth asking.

[markbates]

We’re moving to 1.9 in the next release.

[markbates]

And just to be clear, this has zero to do with Buffalo. I’m just using that repo as an example. Please don’t get hung up on that.

This is about the logic of finding the “latest” version of a v2x repo.

[thepudds]

Is it acceptable to require version 1.9.7+ for buffalo?

As far as I am aware, that it the oldest release that picks up the minimal module awareness that makes older releases able to most easily consume a v2+ module.

[markbates]

Please stop making this about Buffalo. This is about the logic of the resolver.

[thepudds]

Hi @markbates, sorry, we posted at close to the same time.

[thepudds]

Hi @markbates, I'll summarize what I was trying to say, and then leave it to others that are more knowledgeable.

My (possibly incorrect) understanding:

• If there is a properly constructed go.mod file at the root of the tree of Go source files for a package foo, then Go versions 1.9.7+, 1.10.3+, and 1.11 know how to properly interpret a /v2 or /v4 that appears within an import path when importing foo (by interpreting the /v2 or v4 as being a major version).
• But regardless of whether the consumer is Go 1.9.7+, 1.10.3+, 1.11 or vgo, a properly constructed go.mod file for a v4.x.x module includes appending /v4 to the end of the module path in the module directive in the module's go.mod file.
• And perhaps not as relevant here, but Go versions prior to Go 1.9.7 and 1.10.3 have a much more awkward situation regarding consuming a v2 or higher module.

All that said, my understanding might be stale, or perhaps was just wrong to begin with. ;-)