--- title: GitLab Drupal repo: https://gitlab.lakedrops.com/gitlab-ci-cd/pipelines issues: https://gitlab.lakedrops.com/gitlab-ci-cd/pipelines/-/issues tags: - gitlab - ci/cd - pipeline - job --- # GitLab CI for Drupal pipelines This project contains a number of pre-configured tasks for GitLab CI/CD that allow to run very powerful pipelines which contain the following stage order: - [.pre (predefined)](#pre) - [execute](#execute) - [buildprod](#build-prod) - [build (predefined)](#build) - [prepare](#prepare) - [postprocess](#postprocess) - [validate](#validate) - [test (predefined)](#test) - [deploy (predefined)](#deploy) - [finalize](#finalize) - [release](#release-stage) We have a predefined `execute` job called `LakeDrops`, which looks like this: - [execute](#execute) - LakeDrops - [buildprod](#build-prod) - [Build Prod Site](#build-prod-site) - [Build Prod Site NG](#build-prod-site-ng) - [build (predefined)](#build) - [Build Site](#build-site) - [Build Theme](#build-theme) - [Check 4 Outdated Packages](#check-4-outdated-packages) - [Check 4 Security Vulnerability Advisories](#check-4-security-vulnerability-advisories) - [Download DB](#download-db) - [Download DB NG](#download-db-ng) - [prepare](#prepare) - [Import DB](#import-db) - [Import DB NG](#import-db-ng) - [Import Local DB](#import-local-db) - [Update DB](#update-db) - [test](#test) - [Test Code Style](#test-code-style) - [Test PHPUnit](#test-phpunit) - [Test Backstop](#test-backstop) - [Test Cypress E2E](#test-cypress-e2e) - [finalize](#finalize) - [Shut Down Docker Project](#shut-down-docker-project) These pre-configured tasks can easily be included into your Drupal project. A GitLab runner on any host can then be configured to run those pipelines. The following chapters describe the necessary steps and also all the options available for configuration. ## Stages The following chapters describes all stages and their jobs. The files: - [lakedrops.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/lakedrops.yml?ref_type=heads). - [tests/lakedrops.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/tests/lakedrops.yml?ref_type=heads). - [test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads). - [check4updates.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/check4updates.yml?ref_type=heads). ### .pre To prepare and check, if the environment is good to go, we have a `.pre` stage, which contains the following jobs: #### Sanity Checks The sanity check gets important when you are using [Crowdsec](https://www.crowdsec.net/). It contains two little scripts, that check: - if the `crowdsec.settings.yml` is in the config split and - if it is not enabled in the `core.extensions.yml`. Otherwise, it is conflicting with you local settings, since `CrowdSec` is a tool for production sites. Except: - Commit message contains "Merge tag" - Commit message contains "Merge branch" - When the job is triggered by config auto export (CAE) #### Validate Environment This job extends the `.prerequisites` in [test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads). This job executes a script, where several [variables](#variables) are checked and set accordingly. Except: - Commit message contains "Merge tag" - Commit message contains "Merge branch" - When the job is triggered by config auto export (CAE) ### Validate This stage is currently not officially used. ### Execute This stage executes the build itself. ### Build Prod This stage is responsible for building the production website, which we want to finally deploy. #### Build Prod Site First we have to do some preparation. This is done in the `before_script` section. - [.prepareaccess](#prepareaccess) - [.preparecomposerplugins](#preparecomposerplugins) We start a docker image for php with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` In the script section we first update the environment by executing `/usr/local/bin/update-env`. After that we install the production site with `composer install` **without** the tools for development. That is pretty it. We also add `drupal/core-vendor-hardening`. The entire installation is saved as a build artefact called `build-prod` where all `git` files are excluded. #### Build Prod Site NG First we have to do some preparation. This is done in the `before_script` section. - [.prepareaccess](#prepareaccess) - [.preparecomposerplugins](#preparecomposerplugins) We start a docker image for php with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` In the script part we install two of our LakeDrops modules: - [lakedrops/drupal-environment](../../composer/plugin/drupal-environment) - [lakedrops/docker4drupal](../../composer/plugin/d4d) After that we execute: - `composer lakedrops:config` - `composer lakedrops:docker4drupal` The entire installation is saved as a build artefact called `build-prod-ng` where all `git` files are excluded. Rules, when the job should run: - `$DISABLE_DEPLOYMENT_LOCAL` != "1" ### Build Here, we define jobs, which build the Drupal website and does some additional checks. This stage is responsible for the database downloads as well. #### Build Site We start a docker image for `php` with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` After that, we start to build the Drupal site with [Composer](https://getcomposer.org/): Now we execute a `composer install` twice to ensure that all patches get applied correctly. After that we setup our own `LakeDrops` tools by `composer lakedrops:scaffold` and `composer lakedrops:docker4drupal`. The next step is to pull all required [Docker](https://www.docker.com/) images with [Docker Compose](https://docs.docker.com/compose/) and start the services. After everything has started, we create some directories, e.g. for styles, logs and JavaScript. The entire installation is saved as a build artefact called `build` where all `git` files are excluded. Rules, when the job should run: - the tests are enabled by `$TESTSDISABLED` == "no" #### Build Theme This job creates the required files for the styling of the website. The following jobs must run before successfully: - [Build Prod Site](#build-prod-site) #### Check 4 Updates e start a docker image for `php` with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` First we have to do some preparation. This is done in the `before_script` section. - [.prepareaccess](#prepareaccess) - [.preparecomposerplugins](#preparecomposerplugins) The job stores the artefacts in `check4updates`. Rules, when the job should run: - `$CHECKUPDATES` is set - we are on the `develop` branch #### Check 4 Outdated Packages We start a docker image for `php` with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete: - [Check 4 Updates](#check-4-updates) - uses its artefacts - optional - [Build Prod Site](#build-prod-site) - uses its artefacts - optional - [Build Prod Site NG](#build-prod-site-ng) - uses its artefacts - optional - [Build Site](#build-site) - uses its artefacts - optional First we have to do some preparation. This is done in the `before_script` section. - [.prepareaccess](#prepareaccess) - [.preparecomposerplugins](#preparecomposerplugins) First we perform a `composer update` which all dependencies without the development part. Then we check for outdated packages: ```yml composer outdated --minor-only --strict --no-interaction $OUTDATED_EXTRAS composer outdated --patch-only --strict --no-interaction $OUTDATED_EXTRAS ``` The variable `$OUTDATED_EXTRAS` contains the ignored package like: `--ignore=drupal/somemodule --ignore=drupal/another` Rules, when the job must not run: - the commit message contains `SKIP_check4outdated` and `$CHECKUPDATES` != "yes" - the commit message contains "Merge branch 'develop' into 'main'" - the commit message contains "Merge branch 'release' into 'main'" Rules, when the job should run: - `$IGNORE_COMPOSER_EXTENDED_AUDIT` == "0" #### Check 4 Security Vulnerability Advisories We start a docker image for `php` with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete: - [Check 4 Updates](#check-4-updates) - uses its artefacts - optional - [Build Prod Site](#build-prod-site) - uses its artefacts - optional - [Build Prod Site NG](#build-prod-site-ng) - uses its artefacts - optional - [Build Site](#build-site) - uses its artefacts - optional First we have to do some preparation. This is done in the `before_script` section. - [.prepareaccess](#prepareaccess) - [.preparecomposerplugins](#preparecomposerplugins) First we perform a `composer update` which all dependencies without the development part. Then we check for outdated packages: ```yml composer diagnose --no-interaction || true composer audit --no-dev --locked --format=table ``` Rules, when the job must not run: - the commit message contains `SKIP_check4security` and `$CHECKUPDATES` != "yes" - the commit message contains "Merge branch 'develop' into 'main'" - the commit message contains "Merge branch 'release' into 'main'" Rules, when the job should run: - `$IGNORE_COMPOSER_AUDIT` == "0" #### Download DB This job executes a script, that removes the SQL file `$]PROJECT_NAME}.sql`, if it exists already. After that an Ansible script is executed, which dumbs the current Drupal database of the live stage. The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`. Rules, when the job must not run: - `$DISABLE_DEPLOYMENT` != "0" - `$DISABLE_DEPLOYMENT_LOCAL` != "1" - `$DBREQUIRED` == "no" - `$INITIALINSTALL` == "yes" - `$TESTSDISABLED` == "yes" - `$LOCALDBFILE` != "none" Rules, when the job should run: - the tests are enabled by `$DISABLE_CI_TESTS` != "1" #### Download DB NG This is very similar to [Download DB](#download-db), but does not use Ansible. It uses Docker Compose instead to execute the `drush sql:dump` command to dumb the current live database to the file `$]PROJECT_NAME}.sql`. The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`. Rules, when the job must not run: - `$DISABLE_DEPLOYMENT` != "0" - `$DISABLE_DEPLOYMENT_LOCAL` != "0" - `$DBREQUIRED` == "no" - `$INITIALINSTALL` == "yes" - `$TESTSDISABLED` == "yes" - `$LOCALDBFILE` != "none" Rules, when the job should run: - the tests are enabled by `$DISABLE_CI_TESTS` != "1" ### Prepare This stage is used to prepare the system for testing. This stage is also used for internal purpose and for proprietary tools. #### Import DB This job extends `.importdb`, which itself defines the anchor [import_db_default](#import-db-default-anchor). The following jobs must run before successfully: - [Build Site](#build-site) - [Download DB](#download-db) Rules, when the job must not run: - `$DISABLE_DEPLOYMENT` != "0" - `$DISABLE_DEPLOYMENT_LOCAL` != "1" - `$DBREQUIRED` == "no" - `$INITIALINSTALL` == "yes" - `$TESTSDISABLED` == "yes" - `$LOCALDBFILE` != "none" Rules, when the job should run: - the tests are enabled by `$DISABLE_CI_TESTS` != "1" ##### Import DB Default Anchor We start a docker image for `php` with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` A script copies the available SQL file into the PHP container, drops the current DB and imports the file. If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. After that the `UPDATE_DB_COMMAND` gets called as well as a cache reset. #### Import DB NG This job extends `.importdbNG`, which itself defines the anchor [import_db_default](#import-db-default-anchor). The following jobs must run before successfully: - [Build Site](#build-site) - [Download DB NG](#download-db-ng) Rules, when the job must not run: - `$DISABLE_DEPLOYMENT` != "0" - `$DISABLE_DEPLOYMENT_LOCAL` != "0" - `$DBREQUIRED` == "no" - `$INITIALINSTALL` == "yes" - `$TESTSDISABLED` == "yes" - `$LOCALDBFILE` != "none" Rules, when the job should run: - the tests are enabled by `$DISABLE_CI_TESTS` != "1" #### Import Local DB This job extends `.importdbLocal`, which itself defines the anchor [import_db_default](#import-db-default-anchor). The following jobs must run before successfully: - [Build Site](#build-site) A before script is executed, which copies the `${LOCALDBFILE}` to `${PROJECT_NAME}.sql.gz` and unzips that file. Rules, when the job must not run: - `$DBREQUIRED` == "no" - `$TESTSDISABLED` == "yes" - `$LOCALDBFILE` != "none" Rules, when the job should run: - the tests are enabled by `$DISABLE_CI_TESTS` != "1" #### Update DB We start a docker image for php with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. After that the `UPDATE_DB_COMMAND` gets called as well as a cache reset. The following jobs must run before successfully: - [Build Site](#build-site) Rules, when the job must not run: - `$DBREQUIRED` == "yes" - `$DBUPDREQUIRED` == "no" - `$INITIALINSTALL` == "yes" - `$TESTSDISABLED` == "yes" Rules, when the job should run: - the tests are enabled by `$DISABLE_CI_TESTS` != "1" ### Postprocess #### Check 4 Updates Commit We start a docker image for php with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete: - [Check 4 Updates](#check-4-updates) - uses its artefacts - [Check 4 Security Vulnerability Advisories](#check-4-security-vulnerability-advisories) - optional - [Check 4 Outdated Packages](#check-4-outdated-packages) - optional First we have to do some preparation. This is done in the `before_script` section. - [.prepareaccess](#prepareaccess) - [.preparecomposerplugins](#preparecomposerplugins) Rules, when the job should run: - `$CHECKUPDATES` is set - we are on the `develop` branch ### Test After the preparation and the setup of a Drupal database, we can execute tests. This happens in this stage. All tests use the two [Internal Test Stages](#internal-test-stages). #### Test Code Style This job uses [PHP Code Sniffer](../../dev_tools/test.md#php-code-sniffer) and stores the artefacts in `phpcs`. Rules, when the job should run: - the code style tests are enabled by `$DISABLE_CI_TEST_CODESTYLE` != "1" #### Test PHPUnit This job uses [PHP Unit](../../dev_tools/test.md#unitkernelfunctional-testing) and stores the artefacts in `phpunit`. Rules, when the job should run: - the code style tests are enabled by `$DISABLE_CI_TEST_PHPUNIT` != "1" #### Test Backstop This job uses [Backstop](../../dev_tools/test.md#unitkernelfunctional-testing) and stores the artefacts in `backstop`. The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete: - [Build Site](#build-site) - [Build Theme](#build-theme) - uses its artefacts - optional - [Import DB](#import-db) - optional - [Import DB NG](#import-db-ng) - optional - [Import Local DB](#import-local-db) - optional - [Update DB](#update-db) - optional Rules, when the job should run: - the backstop tests are enabled by `$DISABLE_CI_TEST_BACKSTOP` != "1" #### Test Cypress E2E This job uses [Cypress](../../dev_tools/test.md#cypress) and stores the artefacts in `cypresse2e`. The tests are executed in headless mode. The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them to complete: - [Build Site](#build-site) - [Build Theme](#build-theme) - uses its artefacts - optional - [Import DB](#import-db) - optional - [Import DB NG](#import-db-ng) - optional - [Import Local DB](#import-local-db) - optional - [Update DB](#update-db) - optional - [Test Backstop](#test-backstop) - optional Rules, when the job should run: - the cypress tests are enabled by `$DISABLE_CI_TEST_CYPRESSE2E` != "1" #### Internal Test Stages Every test stage needs internal reoccurring definitions. **.test_lakedrops_default** We start a docker image for `php` with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` The following jobs must run before successfully: - [Build Site](#build-site) But this job can start running when the [Build Site](#build-site) has finished, so other test jobs can run in parallel. After running, we copy the logs from the `php` container to the current directory. **.test_lakedrops_theme** The following jobs must run before successfully: - [Build Site](#build-site) - [Build Theme](#build-theme) In the `before_script` part, we copy the `$THEME_CSS_PATH` into the `php` container, if the `$THEME_BUILD` variable ist set to "yes". In `after_script` we remove `$THEME_CSS_PATH` again. #### ### Deploy This stage handles the deployment of the produced build artefact. #### Deploy The following jobs must run before successfully: - [Build Prod Site](#build-prod-site) - [Build Theme](#build-theme) This job executes an [Ansible](https://www.ansible.com/) script, which does the deployment. Rules, when the job must not run: - `$DISABLE_DEPLOYMENT` != "0" - `$DISABLE_DEPLOYMENT_LOCAL` != "1" #### Deploy NG The following jobs must run before successfully: - [Build Prod Site](#build-prod-site) - [Build Prod Site NG](#build-prod-site-ng) - [Build Theme](#build-theme) First we have to do some preparation. This is done in the `before_script` section. - [.prepareaccess](#prepareaccess) This job does not use [Ansible](https://www.ansible.com/). A lot of commands are executed in the script section. Here is a summary: - create the directories, where the site gets deployed to: - app - db - files - redis - backup - sets the ownership and permissions of these files - start all the docker service with [Docker Compose](https://docs.docker.com/compose/) - execute several `drush` commands - create config auto export directory - restart cron jobs To see all the commands, visit [test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads). Rules, when the job must not run: - `$DISABLE_DEPLOYMENT` != "0" - `$CI_COMMIT_BRANCH` != "main" && `$ENFORCE_DEPLOYMENT` != "1" ### Finalize Here we can do cleanups and remove some garbage the build process produces. #### Shut Down Docker Project We start a docker image for php with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` The following jobs must run before successfully: - [Build Site](#build-site) This job simply stops and removes the docker container, which are started for the build by: `docker compose down` ### Release Stage This stage is responsible for creating a new release. #### Release Job This job extends the internal job [.merge](#merge). Rules, when the job should run: - `$CI_COMMIT_MESSAGE` contains `CREATE_RELEASE` - we are on the `develop` branch Rules, when the job must not run: - if the reference is a `tag` - `$CI_COMMIT_MESSAGE` contains `Merge tag` - if it is triggered by config auto export (CAE) - `$CHECKUPDATES` is set - `$TRIGGERTASK` is set ### Internal Stages Here we have some internal stages, which cannot be executed alone. The purpose is to collect repeating execution. #### .prepareaccess This internal job, you find in [mixins.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/general/-/blob/main/mixins.yml?ref_type=heads). Mainly, we care about getting access to: - the host `$CI_SERVER_HOST` with `ssh` via `$SSH_PRIVATE_KEY` - GitLab Login via `$GITLAB_ACCESS_TOKEN` and `$GITLAB_PRIVATE_TOKEN` - Docker configuration via `$DOCKER_AUTH_CONFIG` #### .preparecomposerplugins This internal job, you find in [mixins.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/general/-/blob/main/mixins.yml?ref_type=heads). The following points will be prepared for [Composer](https://getcomposer.org/): - the version >2, if `$DOWNGRADE_COMPOSER` is not set - the configuration `allow-plugins` - the configuration `audit.abandoned`, depend on `$IGNORE_COMPOSER_ABANDONED_AUDIT` - unsupported module versions, which are allowed to use #### .merge First we have to do some preparation. This is done in the `before_script` section. - [.prepareaccess](#prepareaccess) We start a docker image for php with the specifies versions: - `PHP_MAJOR_VERSION` - `PHP_MINOR_VERSION` We create a clean directory and clone the git project `$CI_PROJECT_PATH` for branch `$SOURCE_BRANCH` from `$CI_SERVER_HOST` right into it. After that we execute the [merge](https://gitlab.lakedrops.com/docker/gitlab-drupal-ci/-/blob/main/bin/merge?ref_type=heads) script, which does a few checks and use the `gitlab` tool to merge. Finally, we remove the directory we created above. #### .retry-for-system-issues This little job ste the amount of retries to "2", when: - runner_system_failure - stuck_or_timeout_failure #### .cache_paths All internal jobs concerning cache will use this job here. It defines the paths: - .docker-init/ - assets/ - drush/ - files/ - keys/ - settings/ - vendor/ - web/core/ - web/libraries/ - web/modules/contrib/ - web/profiles/contrib/ - web/sites/ - web/themes/contrib/ - .ahoy.yml - .env - docker-compose.yml