Auto-restart of l3d containers causes boot/login problems on Ubuntu 20 and later
Following this issue, there is a perfect explanation of what's going wrong:
I think I understand what's happening, but don't think there's an easy (or at least not a "generic") solution.
I suspect Ubuntu 20.04 started using $XDG_RUNTIME_DIR for the ssh agent. $XDG_RUNTIME_DIR is specific to each user, and (see the XDG Base Directory Specification) is bound to the user's login session;
The lifetime of the directory MUST be bound to the user being logged in. It MUST be created when the user first logs in and if the user fully logs out the directory MUST be removed.
The problem in this case is that;
The docker daemon is a "system" process, it's not bound to a specific user. The docker service is started before users are logged in (which also is the common situation for a server situation)
However, the container you created;
- Has a restart policy; it will be restarted when docker restarts (and docker is not aware that that specific container assumes a login session is started for user "X")
- While the compose file uses $SSH_AUTH_SOCK to get the correct path for the socket, that environment variable will only be evaluated when the container is created. After that, the container's configuration is immutable. (so even if the ssh-agent is started when the container starts, it could mean it's looking for an old / wrong location)
- The compose file uses the "short-hand" notation to define the bind-mount (:). The short-hand notation will automatically create the path on the host if it's missing. That seemed like a "good idea" at the time, but in hind-sight is causing many issues (we tried deprecating the behavior, but too many people rely on it, so we had to revert that Un-deprecate auto-creation of host directories for mounts moby/moby#21666).
- If the path ($SSH_AUTH_SOCK in this case) isn't found when the container is started, the docker daemon will create the path on the host. Given that it has no knowledge if the path is expected to be a "directory" or a "file", the default ("directory") is assumed, and because the daemon runs as root, the path is created with root as owner. Effectively, this means that when the container is started, and $SSH_AUTH_SOCK is missing (which is the case until the user logs in), that the daemon creates a directory with that name.
As a workaround, you can use the long-form notation to specify the bind-mount; https://docs.docker.com/compose/compose-file/#long-syntax-3 (which is the equivalent of the long-form --mount flag on docker run). The long-form syntax uses a different API, and will not automatically create the path on the host; instead, it will produce an error (which causes the container to fail when starting, until the path in $SSH_AUTH_SOCK is found. With a restart policy set on the container, this will likely mean the container will continue fail (and be restarted) until the user logs in.