Deployment NG: implement DB and files download

For the local development environment we have commands like ahoy live pull* and they use drush commands for download and sync. While drush works remotely in general, file operations in a remote docker container are not yet supported.

We also provide ChatOp commands to trigger pipelines that build that data as artefacts. They will have the same issues.

The CI pipelines require access to the same data too when tests are enabled.

Let's implement a solution that works in remote docker containers and don't require SSH so that all developers can use them, even without access to the host's console.

added To Do label

added Doing label and removed To Do label

assigned to @jurgenhaas

As a quick win for our local dev environment - and therefore maybe even for the pipelines - we can use the docker-compose support from Drush. This requires the following adjustments:

Default alias

Let's define a default alias like this:

local:
  uri: '...'
  docker:
    service: php
    compose:
      options: --project-name [VAR] --file docker-compose.yml --project-directory [HOSTPATH]

Remote aliases

Those should be defined in the same way with the docker specification as for the default. This then requires an extra option --context remote

/usr/local/bin/drush

This should be simplified and call /drupal/vendor/bin/drush with all the given arguments from the command line. If no alias is provided, add the default alias.

/usr/local/bin/docker-compose

This should check for the context argument and put that onto the docker command.

#!/bin/bash

ACTIVE=0
CONTEXT=default
ARGS=''
for arg in "$@"; do
  if [[ $arg == '--context' ]]; then
    ACTIVE=1
  elif [[ ACTIVE -eq 1 ]]; then
    CONTEXT=$arg
    ACTIVE=0
  else
    ARGS="${ARGS} ${arg}"
  fi
done

docker --context $CONTEXT compose $ARGS
EC=$?

exit $EC

Docker context

When starting l3d, create docker contexts for the current project and all of its stages.

mentioned in commit docker/gitlab-drupal-ci@d412256c

• Add Python GitLab client to the images: https://python-gitlab.readthedocs.io/en/stable/index.html
• Use GitLab REST API:
  • https://docs.gitlab.com/ee/api/api_resources.html
  • https://docs.gitlab.com/ee/api/projects.html
  • Search projects: https://gitlab.lakedrops.com/api/v4/projects?search=manegg
  • Get project details: https://gitlab.lakedrops.com/api/v4/projects/508

marked composer/plugin/drupal-development-environment#9 (closed) as a duplicate of this issue

mentioned in issue composer/plugin/drupal-development-environment#9 (closed)

marked this issue as related to composer/plugin/drupal-development-environment#9 (closed)

mentioned in commit 8aad7590

mentioned in commit 43d2bf2c

mentioned in commit a36cfead

The following ChatOps commands are now implemented:

# Collect DB from main site
/KEYWORD run CollectDatabase

# Collect files from main site
/KEYWORD run CollectFiles

# Copy data (db and files) from source branch to target branch
/KEYWORD run CopyDataToStage SOURCEBRANCH TARGETBRANCH

# Reset locales on BRANCH
/KEYWORD run ResetLocales BRANCH

mentioned in commit 54f9b818

mentioned in commit docker/gitlab-drupal-ci@04a85149

mentioned in commit composer/plugin/docker4drupal@2194f618

mentioned in commit docker/l3d@c3661edc

This is now completely implemented. Please update to

L3D v2.5.4
Docker4Drupal v3.7.38

and all the ahoy commands (e.g. a live pull-sql will work again - both for the old and the new deployment generation.

The following new environment variables are required and they should be added to the ~/.env file in the developer's home directory:

GITLAB_URL=https://gitlab.lakedrops.com
GITLAB_PRIVATE_TOKEN=[YOUR_PRIVATE_TOKEN]

Private access tokens can be created here: https://gitlab.lakedrops.com/-/profile/personal_access_tokens

ChatOps commands are now also working in all environments again.

And last but not least, CI pipelines can now also download and import database from the new generation hosting for tests during their build pipelines.

closed

removed Doing label

mentioned in commit c0402317

mentioned in issue #30 (closed)