[BUG] NPM v7 uses SSH instead of an explicit HTTPS for GitHub repos

[uhop]

Current Behavior:

When I use a git repository via an HTTP link NPM "takes liberties" with it, which breaks my build:

$ npm init -y
Wrote to /Users/eugene.lazutkin/Work/temp/package.json:

{
  "name": "temp",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


$ npm i --save https://github.com/uhop/stream-chain.git

added 1 package, and audited 2 packages in 3s

found 0 vulnerabilities

It produces package-lock.json:

{
  "name": "temp",
  "version": "1.0.0",
  "lockfileVersion": 2,
  "requires": true,
  "packages": {
    "": {
      "version": "1.0.0",
      "license": "ISC",
      "dependencies": {
        "stream-chain": "github:uhop/stream-chain"
      }
    },
    "node_modules/stream-chain": {
      "version": "2.2.4",
      "resolved": "git+ssh://git@github.com/uhop/stream-chain.git#459f5a1708c138b6e0abaae4cf103c3488e1e78e",
      "license": "BSD-3-Clause"
    }
  },
  "dependencies": {
    "stream-chain": {
      "version": "git+ssh://git@github.com/uhop/stream-chain.git#459f5a1708c138b6e0abaae4cf103c3488e1e78e",
      "from": "stream-chain@github:uhop/stream-chain"
    }
  }
}

Note that https://github.com/uhop/stream-chain.git was replaced with github:uhop/stream-chain, which is probably OK in this case. But other two links (?) are rewritten from https://github.com/uhop/stream-chain.git to git+ssh://git@github.com/uhop/stream-chain.git, which is clearly bad.

The problem is that a build bot we use in similar situations can access private git repositories using HTTP, but not SSH for security reasons. It fails on an authentication. Rewriting https://github.com/uhop/stream-chain.git to git+ssh://git@github.com/uhop/stream-chain.git is not acceptable for that reasons.

The fix is relatively minor yet unpleasant: we have to replace npm ci with npm i, which takes more time and introduced instabilities with other dependencies.

Expected Behavior:

When running npm ci it should use the original URL with the HTTP authentication instead of SSH.

Steps To Reproduce:

See the description and do the same steps using git repositories (github only?) as dependencies.

Environment:

OS: Mac
Node: 15.7.0
NPM: 7.4.3

[ljharb]

Can you try with npm v7.5.2? I think this has already been fixed.

[uhop]

Just tried with exactly same results.

$ npm --version
7.5.2

package-lock.json is absolutely identical.

[uhop]

I think there is no error in the code.

Just to point it out explicitly:

• The dependency is set as https://github.com/uhop/stream-chain.git — note https:// part of it.
• In package-lock.json it is mentioned three times.
  • Two if them as git+ssh://git@github.com/uhop/stream-chain.git#459f5a1708c138b6e0abaae4cf103c3488e1e78e.
  • Note git+ssh://git@github.com part.
  • It definitely tries to access the repository using ssh rather than https protocol and fails.
    • Why is it important?
      • Those protocols have a different way to authenticate a user.
      • Our build environment is not set up for ssh, yet we can use https.

I don't know if the bug is specific to GitHub: adding https://github.com/uhop/stream-chain.git was transformed into github:uhop/stream-chain (please peruse my bug report above), and who knows what protocol it assumes. But not all repositories support ssh. We clearly have a bug on our hands.

I hope I explained the problem better now. Please do not hesitate to ask, if something is left unclear.

[uhop]

It came to my attention that this is a documented behavior that breaks private repositories and environments without ssh access (from https://blog.npmjs.org/post/626173315965468672/npm-v7-series-beta-release-and-semver-major):

Git dependencies on known git hosts (GitHub, BitBucket, etc.) will always attempt to fetch package contents from the relevant tarball CDNs if possible, falling back to git+ssh for private packages. resolved value in package-lock.json will always reflect the git+ssh url value. Saved value in package.json dependencies will always reflect the canonical shorthand value.

Clearly this is not correct and should be fixed.

[plaa]

This is a breaking change in npm v7 and prevents anyone using even public packages without ssh keys.

We have made a fork of an npm package and host it as a public repo. We refer to it using a github: link. In v6 this github link was populated to package-lock.json as well, which worked fine. In v7 it is converted into git+ssh: which fails in our CI/CD pipeline as it doesn't have any ssh keys configured.

We don't want to add ssh keys to our CI pipeline just to fetch public packages, so I don't see any way we can upgrade. I'd be glad to hear of any workaround.

Specific example:

In package.json referring to https://github.com/Vincit/winston-azure-transport:

    "winston-azure-transport": "github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215"

Diff of package-lock.json between npm v6 and v7:

     "winston-azure-transport": {
-      "version": "github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215",
-      "from": "github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215",```
+      "version": "git+ssh://git@github.com/Vincit/winston-azure-transport.git#ed07d0685ca601638ab8ebcf660128848dd30215",
+      "from": "winston-azure-transport@github:Vincit/winston-azure-transport#ed07d0685ca601638ab8ebcf660128848dd30215",

Our CI now fails with:

npm ERR! Error while executing:
npm ERR! /usr/bin/git ls-remote -h -t ssh://git@github.com/Vincit/winston-azure-transport.git
npm ERR! 
npm ERR! git@github.com: Permission denied (publickey).
npm ERR! fatal: Could not read from remote repository.
npm ERR! 
npm ERR! Please make sure you have the correct access rights
npm ERR! and the repository exists.
npm ERR! 
npm ERR! exited with error code: 128

[isaacs]

This is tricky.

We currently keep the https url if auth is set in the url, because presumably you need that auth to access the repo.

The tricky bit is that using https without auth in the url will trigger a prompt to enter a username and password on the command line, which will fail (or at least be confusing and strange) in many cases where an install with ssh would work fine.

If the repo is public, and a git sha is specified in the spec that we're fetching, then we never hit it via git at all; npm will just fetch the tarball from the appropriate CDN url for the known host. However, if you do not have a git sha in the spec (typically, because there's no lockfile present), then we will have to do a git rev-list to figure out which sha we need to fetch, and that's where it runs into trouble.

Perhaps an approach that both works in build scenarios without ssh keys and for dev machines where ssh keys are present, while minimizing the scenarios where we need to prompt for a username/password for basic auth, would be to do the following for resolving any known-host git spec:

1. If the spec has a git sha, just fetch the tarball from the CDN, and we're done. No git commands needed. If that fails, continue (either a private package or an invalid git sha).
2. If the spec is git+https, and has auth specified, save the full git+https url, and only ever access via https.
3. Attempt to fetch the rev-list via the git+ssh url for the known host/repo. This will fail if ssh keys are not available, but at least will do so without confusingly prompting for user interaction, in the case of private repos where ssh keys are available. If that fails, continue.
4. Attempt to fetch the rev-list via the git+https url for the known host/repo. This will fail if auth is required for a private package. (Note: best to just close stdin immediately, so that it fails as quickly as possible.) But, it will succeed in cases where auth is not required and ssh keys are not available.

Maybe we could also add a config to tell npm which git interaction to prefer, thus swapping the order of (3) and (4) above? --git-prefer=(ssh-first|https-first|ssh-only|https-only), defaulting to ssh-first? Then you could set npm_config_git_prefer=https-only in the build environment, and have it never try ssh at all.

[Tallyb]

Having an all repo config is probably not the best idea, as packages come in all shapes and forms.
I think respecting the installation method provided during the npm install is probably the best approach. If anyone needs ssh or https, they should simply add the package this way.