---
title: GitLab Drupal
repo: https://gitlab.lakedrops.com/gitlab-ci-cd/pipelines
issues: https://gitlab.lakedrops.com/gitlab-ci-cd/pipelines/-/issues
tags:
- gitlab
- ci/cd
- pipeline
- job
---
# GitLab CI for Drupal pipelines

This project contains a number of pre-configured tasks for GitLab CI/CD that
allow to run very powerful pipelines which contain the following stage order:

- [.pre (predefined)](#pre)
- [execute](#execute)
- [buildprod](#build-prod)
- [build (predefined)](#build)
- [prepare](#prepare)
- [postprocess](#postprocess)
- [validate](#validate)
- [test (predefined)](#test)
- [deploy (predefined)](#deploy)
- [finalize](#finalize)
- [release](#release-stage)

We have a predefined `execute` job called `LakeDrops`, which looks like this:

- [execute](#execute)
    - LakeDrops
        - [buildprod](#build-prod)
            - [Build Prod Site](#build-prod-site)
            - [Build Prod Site NG](#build-prod-site-ng)
        - [build (predefined)](#build)
            - [Build Site](#build-site)
            - [Build Theme](#build-theme)
            - [Check 4 Outdated Packages](#check-4-outdated-packages)
            - [Check 4 Security Vulnerability Advisories](#check-4-security-vulnerability-advisories)
            - [Download DB](#download-db)
            - [Download DB NG](#download-db-ng)
        - [prepare](#prepare)
            - [Import DB](#import-db)
            - [Import DB NG](#import-db-ng)
            - [Import Local DB](#import-local-db)
            - [Update DB](#update-db) 
        - [test](#test)
            - [Test Code Style](#test-code-style)
            - [Test PHPUnit](#test-phpunit)
            - [Test Backstop](#test-backstop)
            - [Test Cypress E2E](#test-cypress-e2e)
        - [finalize](#finalize)
            - [Shut Down Docker Project](#shut-down-docker-project)

These pre-configured tasks can easily be included into your Drupal project.
A GitLab runner on any host can then be configured to run those pipelines. The 
following chapters describe the necessary steps and also all the options
available for configuration.

## Stages
The following chapters describes all stages and their jobs. The files:

-  [lakedrops.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/lakedrops.yml?ref_type=heads).
-  [tests/lakedrops.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/tests/lakedrops.yml?ref_type=heads).
-  [test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).
-  [check4updates.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/check4updates.yml?ref_type=heads).

### .pre

To prepare and check, if the environment is good to go, we have a `.pre` stage, which contains the
following jobs:

#### Sanity Checks

The sanity check gets important when you are using [Crowdsec](https://www.crowdsec.net/).
It contains two little scripts, that check:

-    if the `crowdsec.settings.yml` is in the config split and
-    if it is not enabled in the `core.extensions.yml`.

Otherwise, it is conflicting with you local settings, since `CrowdSec`
is a tool for production sites.

Except:

- Commit message contains "Merge tag"
- Commit message contains "Merge branch"
- When the job is triggered by config auto export (CAE)

#### Validate Environment

This job extends the `.prerequisites` in 
[test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).

This job executes a script, where several [variables](#variables) are checked and set accordingly.

Except:

- Commit message contains "Merge tag"
- Commit message contains "Merge branch"
- When the job is triggered by config auto export (CAE)

### Validate

This stage is currently not officially used.

### Execute

This stage executes the build itself.

### Build Prod

This stage is responsible for building the production website, which we want to finally deploy.

#### Build Prod Site

First we have to do some preparation. This is done in the `before_script` section.

- [.prepareaccess](#prepareaccess)
- [.preparecomposerplugins](#preparecomposerplugins)

We start a docker image for php with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

In the script section we first update the environment by executing `/usr/local/bin/update-env`.

After that we install the production site with `composer install` **without** the tools for development. That is pretty 
it. We also add `drupal/core-vendor-hardening`.

The entire installation is saved as a build artefact called `build-prod` where all `git` files are excluded.

#### Build Prod Site NG

First we have to do some preparation. This is done in the `before_script` section.

- [.prepareaccess](#prepareaccess)
- [.preparecomposerplugins](#preparecomposerplugins)

We start a docker image for php with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

In the script part we install two of our LakeDrops modules:

- [lakedrops/drupal-environment](../../composer/plugin/drupal-environment)
- [lakedrops/docker4drupal](../../composer/plugin/d4d)

After that we execute:

- `composer lakedrops:config`
- `composer lakedrops:docker4drupal`

The entire installation is saved as a build artefact called `build-prod-ng` where all `git` files are excluded.

Rules, when the job should run:

- `$DISABLE_DEPLOYMENT_LOCAL` != "1"

### Build

Here, we define jobs, which build the Drupal website and does some additional checks. This stage is
responsible for the database downloads as well.

#### Build Site

We start a docker image for `php` with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

After that, we start to build the Drupal site with [Composer](https://getcomposer.org/):

Now we execute a `composer install` twice to ensure that all patches get applied correctly. After
that we setup our own `LakeDrops` tools by `composer lakedrops:scaffold` and
`composer lakedrops:docker4drupal`.

The next step is to pull all required [Docker](https://www.docker.com/) images 
with [Docker Compose](https://docs.docker.com/compose/) and start the services.

After everything has started, we create some directories, e.g. for styles, logs and JavaScript.

The entire installation is saved as a build artefact called `build` where all `git` files are excluded.

Rules, when the job should run:

- the tests are enabled by `$TESTSDISABLED` == "no"

#### Build Theme

This job creates the required files for the styling of the website. 

The following jobs must run before successfully:

- [Build Prod Site](#build-prod-site)

#### Check 4 Updates

e start a docker image for `php` with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

First we have to do some preparation. This is done in the `before_script` section.

- [.prepareaccess](#prepareaccess)
- [.preparecomposerplugins](#preparecomposerplugins)

The job stores the artefacts in `check4updates`.



Rules, when the job should run:

- `$CHECKUPDATES` is set
- we are on the `develop` branch


#### Check 4 Outdated Packages

We start a docker image for `php` with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
to complete:

- [Check 4 Updates](#check-4-updates)
    - uses its artefacts
    - optional
- [Build Prod Site](#build-prod-site)
    - uses its artefacts
    - optional
- [Build Prod Site NG](#build-prod-site-ng)
    - uses its artefacts
    - optional
- [Build Site](#build-site)
    - uses its artefacts
    - optional

First we have to do some preparation. This is done in the `before_script` section.

- [.prepareaccess](#prepareaccess)
- [.preparecomposerplugins](#preparecomposerplugins)

First we perform a `composer update` which all dependencies without the development part.

Then we check for outdated packages:

```yml
composer outdated --minor-only --strict --no-interaction $OUTDATED_EXTRAS
composer outdated --patch-only --strict --no-interaction $OUTDATED_EXTRAS
```

The variable `$OUTDATED_EXTRAS` contains the ignored package like:
`--ignore=drupal/somemodule --ignore=drupal/another`

Rules, when the job must not run:

- the commit message contains `SKIP_check4outdated` and `$CHECKUPDATES` != "yes"
- the commit message contains "Merge branch 'develop' into 'main'"
- the commit message contains "Merge branch 'release' into 'main'"

Rules, when the job should run:

- `$IGNORE_COMPOSER_EXTENDED_AUDIT` == "0"

#### Check 4 Security Vulnerability Advisories

We start a docker image for `php` with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
to complete:

- [Check 4 Updates](#check-4-updates)
    - uses its artefacts
    - optional
- [Build Prod Site](#build-prod-site)
    - uses its artefacts
    - optional
- [Build Prod Site NG](#build-prod-site-ng)
    - uses its artefacts
    - optional
- [Build Site](#build-site)
    - uses its artefacts
    - optional

First we have to do some preparation. This is done in the `before_script` section.

- [.prepareaccess](#prepareaccess)
- [.preparecomposerplugins](#preparecomposerplugins)

First we perform a `composer update` which all dependencies without the development part.

Then we check for outdated packages:

```yml
composer diagnose --no-interaction || true
composer audit --no-dev --locked --format=table
```

Rules, when the job must not run:

- the commit message contains `SKIP_check4security` and `$CHECKUPDATES` != "yes"
- the commit message contains "Merge branch 'develop' into 'main'"
- the commit message contains "Merge branch 'release' into 'main'"

Rules, when the job should run:

- `$IGNORE_COMPOSER_AUDIT` == "0"

#### Download DB

This job executes a script, that removes the SQL file `$]PROJECT_NAME}.sql`, if it exists already.
After that an Ansible script is executed, which dumbs the current Drupal database of the live stage.

The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`.

Rules, when the job must not run:

- `$DISABLE_DEPLOYMENT` != "0"
- `$DISABLE_DEPLOYMENT_LOCAL` != "1"
- `$DBREQUIRED` == "no"
- `$INITIALINSTALL` == "yes"
- `$TESTSDISABLED` == "yes"
- `$LOCALDBFILE` != "none"

Rules, when the job should run:

- the tests are enabled by `$DISABLE_CI_TESTS` != "1"

#### Download DB NG

This is very similar to [Download DB](#download-db), but does not use Ansible. 
It uses Docker Compose instead to execute the `drush sql:dump` command
to dumb the current live database to the file `$]PROJECT_NAME}.sql`.

The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`.

Rules, when the job must not run:

- `$DISABLE_DEPLOYMENT` != "0"
- `$DISABLE_DEPLOYMENT_LOCAL` != "0"
- `$DBREQUIRED` == "no"
- `$INITIALINSTALL` == "yes"
- `$TESTSDISABLED` == "yes"
- `$LOCALDBFILE` != "none"

Rules, when the job should run:

- the tests are enabled by `$DISABLE_CI_TESTS` != "1"

### Prepare

This stage is used to prepare the system for testing. This stage is also used for internal purpose
and for proprietary tools.

#### Import DB

This job extends `.importdb`, which itself defines the 
anchor [import_db_default](#import-db-default-anchor).

The following jobs must run before successfully:

- [Build Site](#build-site)
- [Download DB](#download-db)

Rules, when the job must not run:

 - `$DISABLE_DEPLOYMENT` != "0"
 - `$DISABLE_DEPLOYMENT_LOCAL` != "1"
 - `$DBREQUIRED` == "no"
 - `$INITIALINSTALL` == "yes"
 - `$TESTSDISABLED` == "yes"
 - `$LOCALDBFILE` != "none"

Rules, when the job should run:

- the tests are enabled by `$DISABLE_CI_TESTS` != "1"

##### Import DB Default Anchor

We start a docker image for `php` with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

A script copies the available SQL file into the PHP container, drops the current DB and imports the file.
If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. After that the
`UPDATE_DB_COMMAND` gets called as well as a cache reset.

#### Import DB NG

This job extends `.importdbNG`, which itself defines the
anchor [import_db_default](#import-db-default-anchor).

The following jobs must run before successfully:

- [Build Site](#build-site)
- [Download DB NG](#download-db-ng)

Rules, when the job must not run:

- `$DISABLE_DEPLOYMENT` != "0"
- `$DISABLE_DEPLOYMENT_LOCAL` != "0"
- `$DBREQUIRED` == "no"
- `$INITIALINSTALL` == "yes"
- `$TESTSDISABLED` == "yes"
- `$LOCALDBFILE` != "none"

Rules, when the job should run:

- the tests are enabled by `$DISABLE_CI_TESTS` != "1"

#### Import Local DB

This job extends `.importdbLocal`, which itself defines the
anchor [import_db_default](#import-db-default-anchor).

The following jobs must run before successfully:

- [Build Site](#build-site)

A before script is executed, which copies the `${LOCALDBFILE}` to `${PROJECT_NAME}.sql.gz` and
unzips that file.

Rules, when the job must not run:

- `$DBREQUIRED` == "no"
- `$TESTSDISABLED` == "yes"
- `$LOCALDBFILE` != "none"

Rules, when the job should run:

- the tests are enabled by `$DISABLE_CI_TESTS` != "1"

#### Update DB

We start a docker image for php with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. 
After that the `UPDATE_DB_COMMAND` gets called as well as a cache reset.

The following jobs must run before successfully:

- [Build Site](#build-site)

Rules, when the job must not run:

- `$DBREQUIRED` == "yes"
- `$DBUPDREQUIRED` == "no"
- `$INITIALINSTALL` == "yes"
- `$TESTSDISABLED` == "yes"

Rules, when the job should run:

- the tests are enabled by `$DISABLE_CI_TESTS` != "1"

### Postprocess

#### Check 4 Updates Commit

We start a docker image for php with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
to complete:

- [Check 4 Updates](#check-4-updates)
    - uses its artefacts
- [Check 4 Security Vulnerability Advisories](#check-4-security-vulnerability-advisories)
    - optional
- [Check 4 Outdated Packages](#check-4-outdated-packages)
    - optional

First we have to do some preparation. This is done in the `before_script` section.

- [.prepareaccess](#prepareaccess)
- [.preparecomposerplugins](#preparecomposerplugins)



Rules, when the job should run:

- `$CHECKUPDATES` is set
- we are on the `develop` branch

### Test

After the preparation and the setup of a Drupal database, we can execute tests. This happens in this stage.

All tests use the two [Internal Test Stages](#internal-test-stages).

#### Test Code Style

This job uses [PHP Code Sniffer](../../dev_tools/test.md#php-code-sniffer) and stores the artefacts in `phpcs`.

Rules, when the job should run:

- the code style tests are enabled by `$DISABLE_CI_TEST_CODESTYLE` != "1"

#### Test PHPUnit

This job uses [PHP Unit](../../dev_tools/test.md#unitkernelfunctional-testing) and stores the artefacts in `phpunit`.

Rules, when the job should run:

- the code style tests are enabled by `$DISABLE_CI_TEST_PHPUNIT` != "1"


#### Test Backstop

This job uses [Backstop](../../dev_tools/test.md#unitkernelfunctional-testing) and stores the artefacts in `backstop`.

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
to complete:

- [Build Site](#build-site)
- [Build Theme](#build-theme)
    - uses its artefacts
    - optional
- [Import DB](#import-db)
    - optional 
- [Import DB NG](#import-db-ng)
    - optional
- [Import Local DB](#import-local-db)
    - optional
- [Update DB](#update-db)
    - optional

Rules, when the job should run:

- the backstop tests are enabled by `$DISABLE_CI_TEST_BACKSTOP` != "1"


#### Test Cypress E2E

This job uses [Cypress](../../dev_tools/test.md#cypress) and stores the artefacts in `cypresse2e`. The tests
are executed in headless mode.

The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
to complete:

- [Build Site](#build-site)
- [Build Theme](#build-theme)
    - uses its artefacts
    - optional
- [Import DB](#import-db)
    - optional
- [Import DB NG](#import-db-ng)
    - optional
- [Import Local DB](#import-local-db)
    - optional
- [Update DB](#update-db)
    - optional
- [Test Backstop](#test-backstop)
    - optional

Rules, when the job should run:

- the cypress tests are enabled by `$DISABLE_CI_TEST_CYPRESSE2E` != "1"

#### Internal Test Stages

Every test stage needs internal reoccurring definitions.

**.test_lakedrops_default**

We start a docker image for `php` with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

The following jobs must run before successfully:

- [Build Site](#build-site)

But this job can start running when the [Build Site](#build-site) has finished, so other
test jobs can run in parallel.

After running, we copy the logs from the `php` container to the current directory.


**.test_lakedrops_theme**

The following jobs must run before successfully:

- [Build Site](#build-site)
- [Build Theme](#build-theme)

In the `before_script` part, we copy the `$THEME_CSS_PATH` into the `php` container, if the
`$THEME_BUILD` variable ist set to "yes".

In `after_script` we remove `$THEME_CSS_PATH` again.

####

### Deploy

This stage handles the deployment of the produced build artefact.

#### Deploy

The following jobs must run before successfully:

- [Build Prod Site](#build-prod-site)
- [Build Theme](#build-theme)

This job executes an [Ansible](https://www.ansible.com/) script, which does the deployment.

Rules, when the job must not run:

- `$DISABLE_DEPLOYMENT` != "0"
- `$DISABLE_DEPLOYMENT_LOCAL` != "1"

#### Deploy NG

The following jobs must run before successfully:

- [Build Prod Site](#build-prod-site)
- [Build Prod Site NG](#build-prod-site-ng)
- [Build Theme](#build-theme)

First we have to do some preparation. This is done in the `before_script` section.

- [.prepareaccess](#prepareaccess)

This job does not use [Ansible](https://www.ansible.com/). A lot of commands are executed in the script section.
Here is a summary:

- create the directories, where the site gets deployed to:
    - app
    - db
    - files
    - redis
    - backup
- sets the ownership and permissions of these files
- start all the docker service with [Docker Compose](https://docs.docker.com/compose/)
- execute several `drush` commands
- create config auto export directory
- restart cron jobs

To see all the commands, visit
[test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).

Rules, when the job must not run:

- `$DISABLE_DEPLOYMENT` != "0"
- `$CI_COMMIT_BRANCH` != "main" && `$ENFORCE_DEPLOYMENT` != "1"

### Finalize

Here we can do cleanups and remove some garbage the build process produces.

#### Shut Down Docker Project

We start a docker image for php with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

The following jobs must run before successfully:

- [Build Site](#build-site)

This job simply stops and removes the docker container, which are started for the build by:

`docker compose down`

### Release Stage

This stage is responsible for creating a new release.

#### Release Job

This job extends the internal job [.merge](#merge).

Rules, when the job should run:

- `$CI_COMMIT_MESSAGE` contains `CREATE_RELEASE`
- we are on the `develop` branch

Rules, when the job must not run:

- if the reference is a `tag`
- `$CI_COMMIT_MESSAGE` contains `Merge tag`
- if it is triggered by config auto export (CAE)
- `$CHECKUPDATES` is set
- `$TRIGGERTASK` is set


### Internal Stages

Here we have some internal stages, which cannot be executed alone. The purpose is to
collect repeating execution.

#### .prepareaccess

This internal job, you find in [mixins.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/general/-/blob/main/mixins.yml?ref_type=heads).

Mainly, we care about getting access to:

- the host `$CI_SERVER_HOST` with `ssh` via `$SSH_PRIVATE_KEY`
- GitLab Login via `$GITLAB_ACCESS_TOKEN` and `$GITLAB_PRIVATE_TOKEN`
- Docker configuration via `$DOCKER_AUTH_CONFIG`


#### .preparecomposerplugins

This internal job, you find in [mixins.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/general/-/blob/main/mixins.yml?ref_type=heads).

The following points will be prepared for [Composer](https://getcomposer.org/):

- the version >2, if `$DOWNGRADE_COMPOSER` is not set
- the configuration `allow-plugins`
- the configuration `audit.abandoned`, depend on `$IGNORE_COMPOSER_ABANDONED_AUDIT`
- unsupported module versions, which are allowed to use

#### .merge

First we have to do some preparation. This is done in the `before_script` section.

- [.prepareaccess](#prepareaccess)

We start a docker image for php with the specifies versions:

- `PHP_MAJOR_VERSION`
- `PHP_MINOR_VERSION`

We create a clean directory and clone the git project `$CI_PROJECT_PATH`
for branch `$SOURCE_BRANCH` from `$CI_SERVER_HOST` right into it.

After that we execute the [merge](https://gitlab.lakedrops.com/docker/gitlab-drupal-ci/-/blob/main/bin/merge?ref_type=heads) 
script, which does a few checks and use the `gitlab` tool to merge.

Finally, we remove the directory we created above.

#### .retry-for-system-issues

This little job ste the amount of retries to "2", when:

- runner_system_failure
- stuck_or_timeout_failure

#### .cache_paths

All internal jobs concerning cache will use this job here. It defines the paths:

- .docker-init/
- assets/
- drush/
- files/
- keys/
- settings/
- vendor/
- web/core/
- web/libraries/
- web/modules/contrib/
- web/profiles/contrib/
- web/sites/
- web/themes/contrib/
- .ahoy.yml
- .env
- docker-compose.yml