Deployment sometimes fails with "LOADING Redis is loading the dataset in memory" error

This is trigger in RedisCacheTagsChecksum.php line 85, and according to this issue, the module won't deal with this. This article explains where it's coming from, and I read this such that Redis might still be loading data from disk while Drupal is already trying to connect.

However, we're already sleeping for 3 seconds trying to prevent this from happening right after restarting docker compose:

$ docker compose exec -u root php chown -R www-data:www-data .
$ docker compose exec -u root php chown -R www-data:www-data /data
$ docker compose exec -u root php chmod -R ug-w .
$ docker compose exec -u root php chmod -R ug+w /data
$ docker compose exec -u root php bash -c 'chmod -R ug+w web/sites/*/files'
$ docker compose exec -u root php bash -c 'chmod -R ug+w web/sites/*/private'
$ docker compose exec -u root php bash -c 'chmod 444 /data/*/*/.htaccess >/dev/null 2>&1 || true'
$ docker compose exec -u root borgmatic chmod -R go-w-r-x /root/.ssh || true
$ if [[ ! -d /data/${CI_COMMIT_REF_NAME}/backup/config/security ]]; then docker compose exec -u root borgmatic borgmatic --init --encryption repokey || true; fi
$ if [[ -f ./crontabs/jobs.ini ]]; then docker compose exec -u root php chown -R www-data:www-data /var/backups/mysql || true; fi
$ if [[ "$INITIAL" == "no" ]]; then sleep 3; drush -y cache-rebuild; fi

Let's see what else we can do.

added Doing label

changed the description

assigned to @jurgenhaas

mentioned in commit docker/gitlab-drupal-ci@ab03d1c2

mentioned in commit cd210504

mentioned in commit 32a04b21

Implemented the wait tool from https://github.com/ufoscout/docker-compose-wait

closed

removed Doing label

For some reason, we've had old dump.rdb files on docker-based hosts for all sites and stages, and that still caused this issue. I've now deleted all those old files and the problem doesn't happen any more. Let's wait and see, if this eventually comes back as a problem.

The problem is easy to reproduce by running this script:

#!/bin/bash

WAIT_HOSTS=redis:6379
dc up -d redis --force-recreate
dc exec -e WAIT_HOSTS="${WAIT_HOSTS}" -it php /wait
drush cr