Images, volumes and networks lost after update to 4.1.0

[westende]

• I have tried with the latest version of Docker Desktop
• I have tried disabling enabled experimental features
• I have uploaded Diagnostics
• Diagnostics ID: 329C902A-C11D-4A46-9F48-8AC730BD7320/20211005123604

Expected behavior

I expected the state (images, volumes and networks) to persist.

Actual behavior

Images, volumes and networks did not persist.

Information

• macOS Version: 10.15.7
• Intel chip or Apple chip: 2,7 GHz Quad-Core Intel Core i7
• Docker Desktop Version: 4.1.0

[brosenburgh]

Same issue for me.

[djs55]

@westende @brosenburgh thanks for the issue. Could you run

% find ~/Library/Containers/com.docker.docker -name "Docker.*" | grep -v log | xargs stat -x
  File: "/Users/dave/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw"
  Size: 63999836160  FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/    dave)  Gid: (   20/   staff)
Device: 1,18   Inode: 53434226    Links: 1
Access: Thu Oct  7 10:54:33 2021
Modify: Wed Oct  6 19:13:32 2021
Change: Wed Oct  6 19:13:32 2021

I'd like to see whether you have multiple Docker.raw or Docker.qcow2 files.

Could you also run

% cat ~/Library/Group\ Containers/group.com.docker/settings.json | grep dataFolder
  "dataFolder": "/Users/dave/Library/Containers/com.docker.docker/Data/vms/0/data",

I'd like to check if the folder is correct.

[brosenburgh]

@djs55, here you go:

Roberts-MacBook-Pro:~ robertrosenburgh$ find ~/Library/Containers/com.docker.docker -name "Docker.*" | grep -v log | xargs stat -x
  File: "/Users/robertrosenburgh/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw"
  Size: 68719476736  FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/robertrosenburgh)  Gid: (   20/   staff)
Device: 1,4   Inode: 8734992018    Links: 1
Access: Wed Oct  6 09:31:30 2021
Modify: Wed Oct  6 09:31:26 2021
Change: Wed Oct  6 09:31:26 2021
  File: "/Users/robertrosenburgh/Library/Containers/com.docker.docker/Data/vms/0/Docker.qcow2"
  Size: 24657133568  FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/robertrosenburgh)  Gid: (   20/   staff)
Device: 1,4   Inode: 140887393    Links: 1
Access: Tue Oct  5 11:38:44 2021
Modify: Tue Oct  5 11:38:44 2021
Change: Tue Oct  5 11:38:44 2021

and...

Roberts-MacBook-Pro:~ robertrosenburgh$ cat ~/Library/Group\ Containers/group.com.docker/settings.json | grep dataFolder
  "dataFolder": "/Users/robertrosenburgh/Library/Containers/com.docker.docker/Data/vms/0",

[westende]

@djs55, hereby:

➜  ~ find ~/Library/Containers/com.docker.docker -name "Docker.*" | grep -v log | xargs stat -x
  File: "/Users/user/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw"
  Size: 68719476736  FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (10395/ user)  Gid: (   20/   staff)
Device: 1,4   Inode: 12960214410    Links: 1
Access: Thu Oct  7 16:32:57 2021
Modify: Thu Oct  7 16:34:13 2021
Change: Thu Oct  7 16:34:13 2021
  File: "/Users/user/Library/Containers/com.docker.docker/Data/vms/0/Docker.qcow2"
  Size: 49096228864  FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (10395/ user)  Gid: (   20/   staff)
Device: 1,4   Inode: 83795336    Links: 1
Access: Tue Oct  5 09:20:27 2021
Modify: Tue Oct  5 09:20:27 2021
Change: Tue Oct  5 09:20:27 2021

➜  ~ cat ~/Library/Group\ Containers/group.com.docker/settings.json | grep dataFolder
  "dataFolder": "/Users/user/Library/Containers/com.docker.docker/Data/vms/0",

[clinis]

I'm having the same problem. But my results show one more Docker.raw file:

$ find ~/Library/Containers/com.docker.docker -name "Docker.*" | grep -v log | xargs stat -x
  File: "/Users/user/Library/Containers/com.docker.docker/Data/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw"
  Size: 68719476736  FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/   user)  Gid: (   20/   staff)
Device: 1,4   Inode: 8673740367    Links: 1
Access: Fri Oct  1 01:55:35 2021
Modify: Sun Jun 20 15:06:48 2021
Change: Sun Jun 20 15:06:48 2021
  File: "/Users/user/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw"
  Size: 68719476736  FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/   user)  Gid: (   20/   staff)
Device: 1,4   Inode: 8677848915    Links: 1
Access: Wed Oct  6 23:14:46 2021
Modify: Wed Oct  6 23:14:36 2021
Change: Wed Oct  6 23:14:36 2021
  File: "/Users/user/Library/Containers/com.docker.docker/Data/vms/0/Docker.qcow2"
  Size: 10405150720  FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  501/   user)  Gid: (   20/   staff)
Device: 1,4   Inode: 9596035    Links: 1
Access: Tue Oct  5 23:00:24 2021
Modify: Tue Oct  5 18:58:25 2021
Change: Tue Oct  5 18:58:25 2021

$ cat ~/Library/Group\ Containers/group.com.docker/settings.json | grep dataFolder
  "dataFolder": "/Users/user/Library/Containers/com.docker.docker/Data/vms/0",

Had a similar problem when updating to v3.4: #5754

[jtowe1]

Same issue for me today
Diag ID: 68CADBA0-BE8B-4C06-9CAF-3FE76895446A/20211011131206

find ~/Library/Containers/com.docker.docker -name "Docker.*" | grep -v log | xargs stat -x        
  File: "/Users/jtowe/Library/Containers/com.docker.docker/Data/vms/0/Docker.raw"
  Size: 63999836160  FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  502/   jtowe)  Gid: (   20/   staff)
Device: 1,4   Inode: 113415333    Links: 1
Access: Mon Oct 11 09:13:40 2021
Modify: Mon Oct 11 09:25:47 2021
Change: Mon Oct 11 09:25:47 2021
  File: "/Users/jtowe/Library/Containers/com.docker.docker/Data/vms/0/Docker.qcow2"
  Size: 28352118784  FileType: Regular File
  Mode: (0644/-rw-r--r--)         Uid: (  502/   jtowe)  Gid: (   20/   staff)
Device: 1,4   Inode: 5924055    Links: 1
Access: Mon Oct 11 08:18:53 2021
Modify: Mon Oct 11 08:18:52 2021
Change: Mon Oct 11 08:18:52 2021

cat ~/Library/Group\ Containers/group.com.docker/settings.json | grep dataFolder      
  "dataFolder": "/Users/jtowe/Library/Containers/com.docker.docker/Data/vms/0",

[djs55]

Thanks all for your help understanding this one.

There's a bug in the latest version which doesn't handle the case where you are using a Docker.qcow2 but your system supports the newer, higher performance Docker.raw. The expected behaviour was to continue using the old qcow2 until the next reset to defaults, when it would switch to raw. Unfortunately the buggy code switched on upgrade, resulting in containers, volumes, networks disappearing (although they still exist in the old file).

There will be a fix in the next release. If you'd like to try a preview build: https://desktop-stage.docker.com/mac/main/amd64/69861/Docker.dmg and https://desktop-stage.docker.com/mac/main/arm64/69861/Docker.dmg

The preview will continue to use the newest file by default, since this contains your latest work. This will be the Docker.raw in your case.

If you'd like to switch back to the Docker.qcow2 with your old containers, volumes and networks, then install the preview, shut it down and:

mv ~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw ~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.raw.backup

This will move the newer Docker.raw out of the way. You should still have a ~/Library/Containers/com.docker.docker/Data/vms/0/data/Docker.qcow2 file in place.

Ensure your ~/Library/Group\ Containers/group.com.docker/settings.json has

  "dataFolder": "/Users/dave/Library/Containers/com.docker.docker/Data/vms/0/data",

(replace the home directory with the expected one!)

and restart Docker. The preview version should detect the old Docker.qcow2 and continue using it. Let me know if this works for you. If not, please upload another set of diagnostics.

Note that the code in the preview won't delete any files (unless you perform a reset) so nothing should be truly lost.

[mat007]

I’m closing this ticket as Desktop 4.1.1 has been released and includes a fix.
Thanks!

[joe0BAB]

See version 4.2.0 for another fix on that.

[RwwL]

@djs55 Thanks very much for your last post (everything on this thread, really), which seems to have pulled me out of horrible world of hurt.

But what happens to me the next time Docker wants to update? Will my containers disappear again and will I be repeating this process? I'm currently on 4.2.0 (70708) .

[westende]

@djs55 Thank you for your explanation.

[docker-robott]

Closed issues are locked after 30 days of inactivity.
This helps our team focus on active issues.

If you have found a problem that seems similar to this, please open a new issue.

Send feedback to Docker Community Slack channels #docker-for-mac or #docker-for-windows.
/lifecycle locked