Skip to content

4.19.0 breaks vpnkit and some networking functionality #6825

Closed
Closed
4.19.0 breaks vpnkit and some networking functionality#6825
Labels
status/acknowledgedversion/4.19.0

Description

@omgftw
omgftw
opened on May 1, 2023
  • I have tried with the latest version of Docker Desktop
  • I have tried disabling enabled experimental features
  • I have uploaded Diagnostics

Expected behavior

vpnkit should start. It is working properly on 4.18.0

Actual behavior

vpnkit fails and some host network functionality no longer functions (ex: reaching the AWS EC2 metadata endpoint from within a container).
I have been able to reproduce this across multiple users and machines (work and personal). Has been tested on x86, M1, and M2 machines.

Information

  • macOS Version: 13.3.1
  • Intel chip or Apple chip: Intel/M1/M2
  • Docker Desktop Version: 4.19.0

Output of /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check

...
[FAIL] DD0014: are the backend processes running? 1 error occurred:
        * com.docker.vpnkit is not running
...
[FAIL] DD0009: is the vpnkit API responding? dial unix vpnkit.diag.sock: connect: connection refused
...
1 : The test: are the backend processes running?
    Failed with: 1 error occurred:
        * com.docker.vpnkit is not running

Steps to reproduce the behavior

No specific steps. On every machine so far it occurs 100% of the time with 4.19.0, across docker and computer restarts. If I can provide any additional information, please let me know.

Activity

  1. christophermclellan commented on May 2, 2023

    @christophermclellan
    christophermclellan
    on May 2, 2023

    Hi @omgftw , sorry about this. In v4.19 we've replaced vpnkit with gvisor for faster networking performance. You can still use vpnkit by adding “networkType”:”vpnkit” to your settings.json file (located in ~/Library/Group Containers/group.com.docker/settings.json). Let us know if there are any further issues.

    cc @djs55

  2. djs55 commented on May 2, 2023

    @djs55
    djs55
    on May 2, 2023
    Contributor

    @omgftw Regarding

    some host network functionality no longer functions (ex: reaching the AWS EC2 metadata endpoint from within a container).

    Could you provide a repro example to demonstrate the problem?

  3. omgftw commented on May 2, 2023

    @omgftw
    omgftw
    on May 2, 2023
    Author

    @djs55 Sure thing.

    Steps to reproduce the behavior

    The EC2 metadata issue can be easily mocked and reproduced as follows.

    • feel free to change the port to something other than 80 for testing. sudo is only needed in the python command due to using port 80.
    • I'm using python below, but any http server listening on the proper ip/port will work.
    • unless you want to background the processes, these commands will need to split across 2 terminal sessions.
    sudo ifconfig lo0 alias 169.254.169.254 # create an interface alias
sudo python3 -m http.server --bind 169.254.169.254 80 # start an http server listening on the metadata IP
docker run --rm alpine/curl curl http://169.254.169.254 # or whatever method you prefer

    cleanup - remove the alias (This is optional since the alias should not persist reboots)

    sudo ifconfig lo0 -alias 169.254.169.254

    4.18.0: works
    4.19.0: curl: (7) Failed to connect to 169.254.169.254 port 80 after 4 ms: Connection refused

    This is actually a pretty major breaking change for us. We, essentially, use a mock IMDS endpoint to authenticate and fetch short-lived IAM credentials to allow our apps to authenticate in dev, so we can use the same auth/permissions mechanisms from dev to prod.

    Hoping this is a gvisor misconfiguration or bug that can be resolved as otherwise we'll have to update hundreds of engineers configurations to switch it back to vpnkit. In that event is there any guarantee for long-term vpnkit support or is this a first pass at deprecation?

  4. djs55 commented on May 2, 2023

    @djs55
    djs55
    on May 2, 2023
    Contributor

    @omgftw thanks for the clear repro steps. It looks like it is a simple gvisor misconfiguration. I'm making a build with a candidate fix and will share it here when it's ready. Sorry for the disruption this is causing!

  5. djs55 commented on May 2, 2023

    @djs55
    djs55
    on May 2, 2023
    Contributor

    @omgftw notarised builds with a candidate fix:

    When I install this build on my machine and run the python HTTP server I see:

    % docker run --rm alpine/curl curl http://169.254.169.254
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0<!DOCTYPE HTML>
<html lang="en">
<head>
<meta charset="utf-8">
<title>Directory listing for /</title>
</head>
<body>
<h1>Directory listing for /</h1>
<hr>
...

  6. omgftw commented on May 2, 2023

    @omgftw
    omgftw
    on May 2, 2023
    Author

    @djs55 That's great to hear and no problem at all. We just advised people internally to hold off on 4.19.0.

    Wow that was fast. I have validated on my end as well and can no longer reproduce the problem. Things are working as expected. Only thing I'd note (which you're probably already aware of) is that the com.docker.diagnose tool should likely be updated to reflect this change.

    All set as far as this issue is concerned. Would you like me to close it? (feel free to close it yourself if you would prefer)

    Really appreciate your help!

  7. djs55 commented on May 2, 2023

    @djs55
    djs55
    on May 2, 2023
    Contributor

    @omgftw thanks for checking the fix. Regarding the com.docker.diagnose issue: there should be a fix for that in 4.20. I propose to leave this issue open and then we'll close it officially when the release with the fix ships. In the meantime let me know if you spot any other networking issues!

  8. ImageMihaelaStoica
    added
    version/4.19.0
    status/acknowledged
    on May 2, 2023

  9. sandstrom commented on May 5, 2023

    @sandstrom
    sandstrom
    on May 5, 2023

    We're also having connectivity issues on 4.19. Works on 4.18.

    • I tried the notarised version above, and it did not resolve this issue for me.

    • I also tried switching to "networkType":"vpnkit" and it didn't help (though I may have missed something when reverting to vpnkit).

    Our scenario

    We're running a caddy instance as a proxy to other containers.

    Here is a slimmed down version of our caddy file, in case it helps.

    Traffic reaches Caddy, but the reverse_proxying doesn't work.

    We get an error from caddy:

    dial tcp 192.168.65.2:5510: connect: connection refused

    Full caddy error message:

    {"level":"error","ts":1683219371.3035526,"logger":"http.log.error.log0","msg":"dial tcp 192.168.65.2:5510: connect: connection refused","request":{"remote_addr":"10.15.0.1:34670","proto":"HTTP/2.0","method":"GET","host":"hello.dev","uri":"/","headers":{"Cache-Control":["no-cache"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36"],"Cookie":["_lfa=LF1.1.d4799e2bdbff4068.1678907466916; _ga=GA1.2.1091812264.1680003830"],"Sec-Ch-Ua":["\"Google Chrome\";v=\"113\", \"Chromium\";v=\"113\", \"Not-A.Brand\";v=\"24\""],"Upgrade-Insecure-Requests":["1"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-GB,en-US;q=0.9,en;q=0.8,sv;q=0.7"],"Pragma":["no-cache"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"hello.dev"}},"duration":0.000671688,"status":502,"err_id":"168rae3tz","err_trace":"reverseproxy.statusError (reverseproxy.go:886)"}

    Caddy file

    A simplified version of our caddy file:

    {
  auto_https disable_redirects
  ocsp_stapling off
}

(shared) {
  log {
    output stdout
    format console
  }

  tls internal {
    on_demand
  }
}

hello.dev {
  import shared

  reverse_proxy /foo* {env.DOCKER_HOST_IP}:5521
  reverse_proxy /bar* {env.DOCKER_HOST_IP}:5531
  reverse_proxy * {env.DOCKER_HOST_IP}:5510
}

  10. joegoggins commented on May 12, 2023

    @joegoggins
    joegoggins
    on May 12, 2023

    I tried installing https://desktop-stage.docker.com/mac/main/arm64/106744/Docker.dmg on my Macbook M1 laptop running Ventura 13.3.1.

    Running /Applications/Docker.app/Contents/MacOS/com.docker.diagnose check shows com.docker.vpnkit is not running.

    When I open ~/Library/Group Containers/group.com.docker/settings.json I see

      "networkType": "gvisor",
  "useVpnkit": true,

    Changing networkType to vpnkit, then quiting and starting docker desktop, makes the diagnostics work again.

    Then I run docker stack deploy and things work initially, I'm able to open https://localhost:3001 locally. i'm running as a single node swarm here.

    When I run docker stack rm though the system reverts to a failed state. After this,

    docker ps returns Error response from daemon: Bad response from Docker engine

    Then when I re-run diagnostics, it errors like this (below) and fatally exits docker, so next run of docker ps returns Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

    Starting diagnostics

[PASS] DD0027: is there available disk space on the host?
[PASS] DD0028: is there available VM disk space?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[FAIL] DD0004: is the Docker engine running? Get "http://ipc/docker": EOF
[2023-05-12T15:58:21.464148000Z][com.docker.diagnose][I] ipc.NewClient: bf8f3295-com.docker.diagnose -> lifecycle-server.sock VMDockerdAPI
[2023-05-12T15:58:21.464306000Z][com.docker.diagnose][I] (75238f29) bf8f3295-com.docker.diagnose C->S VMDockerdAPI GET /docker
[2023-05-12T15:58:21.466498000Z][com.docker.diagnose][W] (75238f29) bf8f3295-com.docker.diagnose C<-S NoResponse GET /docker (2.18775ms): Get "http://ipc/docker": EOF
[2023-05-12T15:58:21.466694000Z][com.docker.diagnose][I] (75238f29-1) bf8f3295-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-05-12T15:58:21.469138000Z][com.docker.diagnose][W] (75238f29-1) bf8f3295-com.docker.diagnose C<-S NoResponse GET /ping (2.442333ms): Get "http://ipc/ping": EOF
[2023-05-12T15:58:22.470633000Z][com.docker.diagnose][I] (75238f29-2) bf8f3295-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-05-12T15:58:22.473932000Z][com.docker.diagnose][W] (75238f29-2) bf8f3295-com.docker.diagnose C<-S NoResponse GET /ping (3.396167ms): Get "http://ipc/ping": EOF
[2023-05-12T15:58:23.474808000Z][com.docker.diagnose][I] (75238f29-3) bf8f3295-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-05-12T15:58:23.479509000Z][com.docker.diagnose][W] (75238f29-3) bf8f3295-com.docker.diagnose C<-S NoResponse GET /ping (4.702083ms): Get "http://ipc/ping": EOF
[2023-05-12T15:58:24.481745000Z][com.docker.diagnose][I] (75238f29-4) bf8f3295-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-05-12T15:58:24.487565000Z][com.docker.diagnose][W] (75238f29-4) bf8f3295-com.docker.diagnose C<-S NoResponse GET /ping (5.810834ms): Get "http://ipc/ping": EOF
[2023-05-12T15:58:25.492241000Z][com.docker.diagnose][I] (75238f29-5) bf8f3295-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-05-12T15:58:25.495886000Z][com.docker.diagnose][W] (75238f29-5) bf8f3295-com.docker.diagnose C<-S NoResponse GET /ping (3.644792ms): Get "http://ipc/ping": EOF
[2023-05-12T15:58:26.497771000Z][com.docker.diagnose][I] (75238f29-6) bf8f3295-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-05-12T15:58:26.503528000Z][com.docker.diagnose][W] (75238f29-6) bf8f3295-com.docker.diagnose C<-S NoResponse GET /ping (5.739834ms): Get "http://ipc/ping": EOF
[2023-05-12T15:58:27.505617000Z][com.docker.diagnose][I] (75238f29-7) bf8f3295-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-05-12T15:58:27.509615000Z][com.docker.diagnose][W] (75238f29-7) bf8f3295-com.docker.diagnose C<-S NoResponse GET /ping (4.003041ms): Get "http://ipc/ping": EOF
[2023-05-12T15:58:28.511542000Z][com.docker.diagnose][I] (75238f29-8) bf8f3295-com.docker.diagnose C->S VMDockerdAPI GET /ping
[2023-05-12T15:58:28.523601000Z][com.docker.diagnose][W] (75238f29-8) bf8f3295-com.docker.diagnose C<-S NoResponse GET /ping (12.069583ms): Get "http://ipc/ping": EOF

[PASS] DD0015: are the binary symlinks installed?
[FAIL] DD0031: does the Docker API work? error during connect: Get "http://docker.raw.sock/v1.24/containers/json": EOF
[PASS] DD0013: is the $PATH ok?
Error response from daemon: Bad response from Docker engine
[FAIL] DD0003: is the Docker CLI working? exit status 1
[FAIL] DD0038: is the connection to Docker working? HTTP GET https://login.docker.com: Get "https://login.docker.com": proxyconnect tcp: dial unix httpproxy.sock: connect: no such file or directory
[PASS] DD0014: are the backend processes running?
[PASS] DD0007: is the backend responding?
[PASS] DD0008: is the native API responding?
[PASS] DD0009: is the vpnkit API responding?
[PASS] DD0010: is the Docker API proxy responding?
[SKIP] DD0030: is the image access management authorized?
[PASS] DD0033: does the host have Internet access?
[PASS] DD0018: does the host support virtualization?
[PASS] DD0001: is the application running?
[PASS] DD0017: can a VM be started?
[PASS] DD0016: is the LinuxKit VM running?
[PASS] DD0011: are the LinuxKit services running?
[WARN] DD0004: is the Docker engine running? Get "http://ipc/docker": EOF
[PASS] DD0015: are the binary symlinks installed?
[WARN] DD0031: does the Docker API work? error during connect: Get "http://docker.raw.sock/v1.24/containers/json": EOF
[WARN] DD0032: do Docker networks overlap with host IPs? error during connect: Get "http://docker.raw.sock/v1.24/networks": EOF

    Oddly, when I start Docker again, deploy and destroy my stack it does not crash and diagnostics pass now. For now, it looks like I can do some actual development without Docker Desktop crashing. Will report back if I see other issues come up.

    ...and the Error response from daemon: Bad response from Docker engine error is back. Since reverting to 4.18 didn't help either, I'm going back to known good state on 4.12 and will stay tuned to this issue for a fix in a future release.

  11. eararipe commented on May 15, 2023

    @eararipe
    eararipe
    on May 15, 2023

    Hi, I'm having a problem that might be similar. I have an app that binds to port 8081 and will not start on a hackintosh with Docker Desktop 4.19 and Ventura 13.3.1.

    It works on another hackintosh and on my MacBook. They are all Intel machines.

    The affected computer has a RTL8125 nic. It's working on 4.18 and changing network to vpnkit or using @djs55 release candidate did not work. Can I help in anyway?

    edit: I noticed that on the problem computer the port is show as 0.0.0.0:8081, while on the others it's localhost:8081

  12. djs55 commented on May 15, 2023

    @djs55
    djs55
    on May 15, 2023
    Contributor

    Thanks for the reports. I notice env.DOCKER_HOST_IP in one of the examples above. I'm not sure where this environment variable is defined, but the best way to access the host is via the DNS name host.docker.internal. In 4.19 the IP address changed:

    > docker run -it alpine
/ # ping host.docker.internal
PING host.docker.internal (192.168.65.254): 56 data bytes
64 bytes from 192.168.65.254: seq=0 ttl=62 time=3.080 ms
^C
--- host.docker.internal ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 3.080/3.080/3.080 ms

    Can you use the DNS name instead of an IP? This ensures the value will be correct even if the internal IP range is changed in the settings UI.

  13. sandstrom commented on May 15, 2023

    @sandstrom
    sandstrom
    on May 15, 2023

    In our case DOCKER_HOST_IP was hard-coded to 192.168.65.2. We'll investigate if the changed IP might have been the reason.

  14. davem-foreflight commented on May 21, 2023

    @davem-foreflight
    davem-foreflight
    on May 21, 2023

    Just ran into this with 4.19 on Friday... Burned a day because docker was my last suspected reason for a metadata lookup to fail. :^).

  15. lorenrh commented on May 30, 2023

    @lorenrh
    lorenrh
    on May 30, 2023
    Member

    Closing this issue because a fix has been released in Docker Desktop 4.20. See the release notes for more details.

  16. Imagelorenrh
    closed this as completedon May 30, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    status/acknowledgedversion/4.19.0

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions