/usr/bin/toolbox linked against glibc-2.32 doesn't run on older glibc

[juanje]

Describe the bug

Usually, you can run another Fedora release with toolbox by doing:

# On Fedora 32
$ toolbox create --release 31
Created container: fedora-toolbox-31
Enter with: toolbox enter --release 31
$ toolbox enter --release 31

But at Fedora Rawhide you get the following error:

# On Fedora 33
$ toolbox create --release 32
Created container: fedora-toolbox-32
Enter with: toolbox enter --release 32
$ toolbox enter --release 32
Error: invalid entry point PID of container fedora-toolbox-32

The full debug output:

$ toolbox -v enter -r 32
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/local/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user vagrant 
DEBU TOOLBOX_PATH is /usr/local/bin/toolbox       
DEBU Toolbox config directory is /home/vagrant/.config/toolbox 
DEBU Current Podman version is 2.1.0-dev          
DEBU Old Podman version is 2.1.0-dev              
DEBU Migration not needed: Podman version 2.1.0-dev is unchanged 
DEBU Resolving container and image names          
DEBU Container: ''                                
DEBU Image: ''                                    
DEBU Release: '32'                                
DEBU Resolved container and image names           
DEBU Container: 'fedora-toolbox-32'               
DEBU Image: 'fedora-toolbox:32'                   
DEBU Release: '32'                                
DEBU Checking if container fedora-toolbox-32 exists 
DEBU Calling org.freedesktop.Flatpak.SessionHelper.RequestSession 
DEBU Starting container fedora-toolbox-32         
DEBU Inspecting entry point of container fedora-toolbox-32 
DEBU Entry point PID is a float64                 
DEBU Entry point of container fedora-toolbox-32 is toolbox (PID=0) 
Error: invalid entry point PID of container fedora-toolbox-32

Steps how to reproduce the behaviour

1. Enter into a Fedora Rawhide/33 system.
2. Compile and install the current Toolbox versions from the source.
3. Create a new container from the previoues release (32) with:

$ toolbox create --release 32

4. Try to enter at that container by typing:

$ toolbox enter --release 32

5. It fails.

Expected behaviour
That toolbox would enter the container normally.

Actual behaviour
It fails to perform enter on the container.

Output of toolbox --version (v0.0.90+)
toolbox version 0.0.93

Toolbox package info (rpm -q toolbox)
It was installed from the sources, no toolbox package installed.

Output of podman version

Version:      2.1.0-dev
API Version:  1
Go Version:   go1.15rc2
Built:        Thu Jan  1 00:00:00 1970
OS/Arch:      linux/amd64

Podman package info (rpm -q podman)
podman-2.1.0-0.169.dev.git162625f.fc33.x86_64`

Info about your OS
I tested in a virtual machine with Vagrant. The OS is Fedora 33.
It was a fresh installation from today (August 14th) and with all the packages updated.

Additional context
I tried with the releases 31 and 32. The release 33 (the same that the host) was working fine. Well, apart for the bug: #523
Also, I tied the same at another VM with Fedora 32 and it worked fine. I tried at that F32 VM with the releases 29, 31, 32 and 33, with no problems.
I notice though, that the entry point PID, the one from the error, was different. At the Rawhide system the PID was always 0, but at my system (Silverblue 32) and the VM (Fedora 32) were always a non-zero value. Something like PID=32612 and such.

For example (inside Fedora 32 VM):

$ toolbox -v enter -r 29
DEBU Running as real user ID 1000                 
DEBU Resolved absolute path to the executable as /usr/local/bin/toolbox 
DEBU Running on a cgroups v2 host                 
DEBU Checking if /etc/subgid and /etc/subuid have entries for user vagrant 
DEBU TOOLBOX_PATH is /usr/local/bin/toolbox       
DEBU Toolbox config directory is /home/vagrant/.config/toolbox 
DEBU Current Podman version is 2.0.2              
DEBU Old Podman version is 2.0.2                  
...
DEBU Starting container fedora-toolbox-29         
DEBU Inspecting entry point of container fedora-toolbox-29 
DEBU Entry point PID is a float64                 
DEBU Entry point of container fedora-toolbox-29 is toolbox (PID=33068) 
DEBU Waiting for container fedora-toolbox-29 to finish initializing 
...
DEBU --                                           
DEBU -c                                           
DEBU exec "$@"                                    
DEBU /bin/sh                                      
DEBU /bin/bash                                    
DEBU -l

[debarshiray]

DEBU Entry point of container fedora-toolbox-32 is toolbox (PID=0)
Error: invalid entry point PID of container fedora-toolbox-32

This looks like the container failed to start.

Once you have attempted to enter and it failed, could you please try this command and post the logs from it:

$ podman start --attach fedora-toolbox-32

[juanje]

I did and this is the error message:

[vagrant@ci-node-33 ~]$ podman start --attach fedora-toolbox-32
toolbox: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by toolbox)

Here is with the debug output:

[vagrant@ci-node-33 ~]$ podman --log-level debug start --attach fedora-toolbox-32
INFO[0000] podman filtering at log level debug          
DEBU[0000] Called start.PersistentPreRunE(podman --log-level debug start --attach fedora-toolbox-32) 
DEBU[0000] Ignoring libpod.conf EventsLogger setting "/home/vagrant/.config/containers/containers.conf". Use "journald" if you want to change this setting and remove libpod.conf files. 
DEBU[0000] Reading configuration file "/usr/share/containers/containers.conf" 
DEBU[0000] Merged system config "/usr/share/containers/containers.conf": &{Containers:{Devices:[] Volumes:[] ApparmorProfile:containers-default-0.18.0 Annotations:[] CgroupNS:private Cgroups:enabled DefaultCapabilities:[CAP_AUDIT_WRITE CAP_CHOWN CAP_DAC_OVERRIDE CAP_FOWNER CAP_FSETID CAP_KILL CAP_MKNOD CAP_NET_BIND_SERVICE CAP_NET_RAW CAP_SETFCAP CAP_SETGID CAP_SETPCAP CAP_SETUID CAP_SYS_CHROOT] DefaultSysctls:[] DefaultUlimits:[] DefaultMountsFile: DNSServers:[] DNSOptions:[] DNSSearches:[] EnableLabeling:true Env:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] EnvHost:false HTTPProxy:false Init:false InitPath: IPCNS:private LogDriver:k8s-file LogSizeMax:-1 NetNS:slirp4netns NoHosts:false PidsLimit:2048 PidNS:private SeccompProfile:/usr/share/containers/seccomp.json ShmSize:65536k TZ: Umask:0022 UTSNS:private UserNS:host UserNSSize:65536} Engine:{CgroupCheck:true CgroupManager:systemd ConmonEnvVars:[PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin] ConmonPath:[/usr/libexec/podman/conmon /usr/local/libexec/podman/conmon /usr/local/lib/podman/conmon /usr/bin/conmon /usr/sbin/conmon /usr/local/bin/conmon /usr/local/sbin/conmon /run/current-system/sw/bin/conmon] DetachKeys:ctrl-p,ctrl-q EnablePortReservation:true Env:[] EventsLogFilePath:/run/user/1000/libpod/tmp/events/events.log EventsLogger:file HooksDir:[/usr/share/containers/oci/hooks.d] ImageDefaultTransport:docker:// InfraCommand:/pause InfraImage:k8s.gcr.io/pause:3.2 InitPath:/usr/libexec/podman/catatonit LockType:shm Namespace: NetworkCmdPath: NoPivotRoot:false NumLocks:2048 OCIRuntime:/usr/bin/crun OCIRuntimes:map[crun:[/usr/bin/crun /usr/sbin/crun /usr/local/bin/crun /usr/local/sbin/crun /sbin/crun /bin/crun /run/current-system/sw/bin/crun] kata:[/usr/bin/kata-runtime /usr/sbin/kata-runtime /usr/local/bin/kata-runtime /usr/local/sbin/kata-runtime /sbin/kata-runtime /bin/kata-runtime /usr/bin/kata-qemu /usr/bin/kata-fc] runc:[/usr/bin/runc /usr/sbin/runc /usr/local/bin/runc /usr/local/sbin/runc /sbin/runc /bin/runc /usr/lib/cri-o-runc/sbin/runc /run/current-system/sw/bin/runc]] PullPolicy:missing Remote:false RemoteURI: RemoteIdentity: ActiveService: ServiceDestinations:map[] RuntimePath:[] RuntimeSupportsJSON:[crun runc] RuntimeSupportsNoCgroups:[crun] RuntimeSupportsKVM:[kata kata-runtime kata-qemu kata-fc] SetOptions:{StorageConfigRunRootSet:false StorageConfigGraphRootSet:false StorageConfigGraphDriverNameSet:false StaticDirSet:false VolumePathSet:false TmpDirSet:false} SignaturePolicyPath:/etc/containers/policy.json SDNotify:false StateType:3 StaticDir:/home/vagrant/.local/share/containers/storage/libpod StopTimeout:10 TmpDir:/run/user/1000/libpod/tmp VolumePath:/home/vagrant/.local/share/containers/storage/volumes} Network:{CNIPluginDirs:[/usr/libexec/cni /usr/lib/cni /usr/local/lib/cni /opt/cni/bin] DefaultNetwork:podman NetworkConfigDir:/home/vagrant/.config/cni/net.d}} 
DEBU[0000] Using conmon: "/usr/bin/conmon"              
DEBU[0000] Initializing boltdb state at /home/vagrant/.local/share/containers/storage/libpod/bolt_state.db 
DEBU[0000] Using graph driver overlay                   
DEBU[0000] Using graph root /home/vagrant/.local/share/containers/storage 
DEBU[0000] Using run root /run/user/1000/containers     
DEBU[0000] Using static dir /home/vagrant/.local/share/containers/storage/libpod 
DEBU[0000] Using tmp dir /run/user/1000/libpod/tmp      
DEBU[0000] Using volume path /home/vagrant/.local/share/containers/storage/volumes 
DEBU[0000] Set libpod namespace to ""                   
DEBU[0000] [graphdriver] trying provided driver "overlay" 
DEBU[0000] overlay: mount_program=/usr/bin/fuse-overlayfs 
DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false 
DEBU[0000] Initializing event backend file              
DEBU[0000] using runtime "/usr/bin/runc"                
DEBU[0000] using runtime "/usr/bin/crun"                
WARN[0000] Error initializing configured OCI runtime kata: no valid executable found for OCI runtime kata: invalid argument 
DEBU[0000] using runtime "/usr/bin/crun"                
INFO[0000] Setting parallel job count to 7              
DEBU[0000] overlay: mount_data=lowerdir=/home/vagrant/.local/share/containers/storage/overlay/l/55ZKQ2LAMO7JLCJ4FVOE6E6TBG:/home/vagrant/.local/share/containers/storage/overlay/l/2SIOJSIO6LJDSZJHEL6FUA34V2,upperdir=/home/vagrant/.local/share/containers/storage/overlay/3109f3facec37f0bec076d921731107d23a36c918a3eb24bb03685f10179801f/diff,workdir=/home/vagrant/.local/share/containers/storage/overlay/3109f3facec37f0bec076d921731107d23a36c918a3eb24bb03685f10179801f/work,context="system_u:object_r:container_file_t:s0:c14,c464" 
DEBU[0000] mounted container "e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e" at "/home/vagrant/.local/share/containers/storage/overlay/3109f3facec37f0bec076d921731107d23a36c918a3eb24bb03685f10179801f/merged" 
DEBU[0000] Created root filesystem for container e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e at /home/vagrant/.local/share/containers/storage/overlay/3109f3facec37f0bec076d921731107d23a36c918a3eb24bb03685f10179801f/merged 
DEBU[0000] /etc/system-fips does not exist on host, not mounting FIPS mode secret 
DEBU[0000] Setting CGroups for container e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e to user.slice:libpod:e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e 
DEBU[0000] set root propagation to "rslave"             
DEBU[0000] reading hooks from /usr/share/containers/oci/hooks.d 
DEBU[0000] Created OCI spec for container e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e at /home/vagrant/.local/share/containers/storage/overlay-containers/e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e/userdata/config.json 
DEBU[0000] /usr/bin/conmon messages will be logged to syslog 
DEBU[0000] running conmon: /usr/bin/conmon               args="[--api-version 1 -c e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e -u e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e -r /usr/bin/crun -b /home/vagrant/.local/share/containers/storage/overlay-containers/e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e/userdata -p /run/user/1000/containers/overlay-containers/e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e/userdata/pidfile -n fedora-toolbox-32 --exit-dir /run/user/1000/libpod/tmp/exits --socket-dir-path /run/user/1000/libpod/tmp/socket -s -l k8s-file:/home/vagrant/.local/share/containers/storage/overlay-containers/e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e/userdata/ctr.log --log-level debug --syslog --conmon-pidfile /run/user/1000/containers/overlay-containers/e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e/userdata/conmon.pid --exit-command /usr/bin/podman --exit-command-arg --root --exit-command-arg /home/vagrant/.local/share/containers/storage --exit-command-arg --runroot --exit-command-arg /run/user/1000/containers --exit-command-arg --log-level --exit-command-arg error --exit-command-arg --cgroup-manager --exit-command-arg systemd --exit-command-arg --tmpdir --exit-command-arg /run/user/1000/libpod/tmp --exit-command-arg --runtime --exit-command-arg /usr/bin/crun --exit-command-arg --storage-driver --exit-command-arg overlay --exit-command-arg --storage-opt --exit-command-arg overlay.mount_program=/usr/bin/fuse-overlayfs --exit-command-arg --events-backend --exit-command-arg file --exit-command-arg container --exit-command-arg cleanup --exit-command-arg e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e]"
INFO[0000] Running conmon under slice user.slice and unitName libpod-conmon-e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e.scope 
[conmon:d]: failed to write to /proc/self/oom_score_adj: Permission denied

DEBU[0000] Received: 1684                               
INFO[0000] Got Conmon PID as 1681                       
DEBU[0000] Created container e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e in OCI runtime 
DEBU[0000] Attaching to container e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e 
DEBU[0000] connecting to socket /run/user/1000/libpod/tmp/socket/e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e/attach 
DEBU[0000] Starting container e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e with command [toolbox --verbose init-container --home /home/vagrant --monitor-host --shell /bin/bash --uid 1000 --user vagrant] 
DEBU[0000] Started container e54f97de47f50f5ffc98d514629170dce4decd308861aee4e6bccc017abfd24e 
DEBU[0000] Enabling signal proxying                     
toolbox: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by toolbox)
DEBU[0000] Called start.PersistentPostRunE(podman --log-level debug start --attach fedora-toolbox-32) 

[martymichal]

Yep, this is not good. Basically some library pulled in by the Toolbox code uses C code requiring the use of cgo. When cgo is used during the build and a libc library is found on the system, the binary is dynamically linked to the libc library. This is normally ok but since Toolbox is used as the entry-point of containers, different versions of libc (resp. glibc) can break it. I have a proposal for a fix (#531) but I need to discuss it more with @debarshiray.

[debarshiray]

Yes, as @HarryMichal mentioned, this is the problematic part:

[vagrant@ci-node-33 ~]$ podman start --attach fedora-toolbox-32
toolbox: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by toolbox)

It means that a /usr/bin/toolbox binary linked against glibc-2.32 from Fedora 33 on the host, doesn't work with an older glibc from older Fedora releases inside the container.

We end up in this situation because we bind mount /usr/bin/toolbox from the host into the container so that it can be used as the container's entry-point. This worked fine in the past when Toolbox was implemented in POSIX shell, and it would be picked up the environment's /bin/sh. However, it doesn't work so well when it's an ELF binary.

[debarshiray]

I grabbed the Fedora 33 binary, unpacked it and poked at it a bit.

$ rpm2cpio ./toolbox-0.0.93-2.fc33.x86_64.rpm | cpio -idmv
...
...
$ objdump -T ./usr/bin/toolbox | grep GLIBC_2.32
0000000000000000      DO *UND*	0000000000000000  GLIBC_2.32  pthread_sigmask

Looks like there's a new implementation of pthread_sigmask in glibc-2.32.

[debarshiray]

I found a way to tell the Go toolchain to avoid the new version of the pthread_sigmask symbol from glibc-2.32. See: #534

Testing welcome.

[debarshiray]

Closing.

Please feel free to leave a comment on the PRs or here if you think that it's still broken.

Thanks for the testing, by the way. Much appreciated!