Document stages and jobs.

docs/gitlab/drupal/index.md

@@ -7,776 +7,10 @@ tags:
 - ci/cd
 - drupal
 ---
-# GitLab CI for Drupal pipelines
 
-This project contains a number of pre-configured tasks for GitLab CI/CD that
-allow to run very powerful pipelines which contain the following stage order:
+# Configuration
 
-- [.pre (predefined)](#pre)
-- [execute](#execute)
-- [buildprod](#build-prod)
-- [build (predefined)](#build)
-- [prepare](#prepare)
-- [postprocess](#postprocess)
-- [validate](#validate)
-- [test (predefined)](#test)
-- [deploy (predefined)](#deploy)
-- [finalize](#finalize)
-- [release](#release-stage)
-
-We have a predefined `execute` job called `LakeDrops`, which looks like this:
-
-- [execute](#execute)
-    - LakeDrops
-        - [buildprod](#build-prod)
-            - [Build Prod Site](#build-prod-site)
-            - [Build Prod Site NG](#build-prod-site-ng)
-        - [build (predefined)](#build)
-            - [Build Site](#build-site)
-            - [Build Theme](#build-theme)
-            - [Check 4 Outdated Packages](#check-4-outdated-packages)
-            - [Check 4 Security Vulnerability Advisories](#check-4-security-vulnerability-advisories)
-            - [Download DB](#download-db)
-            - [Download DB NG](#download-db-ng)
-        - [prepare](#prepare)
-            - [Import DB](#import-db)
-            - [Import DB NG](#import-db-ng)
-            - [Import Local DB](#import-local-db)
-            - [Update DB](#update-db) 
-        - [test](#test)
-            - [Test Code Style](#test-code-style)
-            - [Test PHPUnit](#test-phpunit)
-            - [Test Backstop](#test-backstop)
-            - [Test Cypress E2E](#test-cypress-e2e)
-        - [finalize](#finalize)
-            - [Shut Down Docker Project](#shut-down-docker-project)
-
-These pre-configured tasks can easily be included into your Drupal project.
-A GitLab runner on any host can then be configured to run those pipelines. The 
-following chapters describe the necessary steps and also all the options
-available for configuration.
-
-## Stages
-The following chapters describes all stages and their jobs. The files:
-
--  [lakedrops.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/lakedrops.yml?ref_type=heads).
--  [tests/lakedrops.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/tests/lakedrops.yml?ref_type=heads).
--  [test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).
--  [check4updates.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/check4updates.yml?ref_type=heads).
-
-### .pre
-
-To prepare and check, if the environment is good to go, we have a `.pre` stage, which contains the
-following jobs:
-
-#### Sanity Checks
-
-The sanity check gets important when you are using [Crowdsec](https://www.crowdsec.net/).
-It contains two little scripts, that check:
-
--    if the `crowdsec.settings.yml` is in the config split and
--    if it is not enabled in the `core.extensions.yml`.
-
-Otherwise, it is conflicting with you local settings, since `CrowdSec`
-is a tool for production sites.
-
-Except:
-
-- Commit message contains "Merge tag"
-- Commit message contains "Merge branch"
-- When the job is triggered by config auto export (CAE)
-
-#### Validate Environment
-
-This job extends the `.prerequisites` in 
-[test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).
-
-This job executes a script, where several [variables](#variables) are checked and set accordingly.
-
-Except:
-
-- Commit message contains "Merge tag"
-- Commit message contains "Merge branch"
-- When the job is triggered by config auto export (CAE)
-
-### Validate
-
-This stage is currently not officially used.
-
-### Execute
-
-This stage executes the build itself.
-
-### Build Prod
-
-This stage is responsible for building the production website, which we want to finally deploy.
-
-#### Build Prod Site
-
-First we have to do some preparation. This is done in the `before_script` section.
-
-- [.prepareaccess](#prepareaccess)
-- [.preparecomposerplugins](#preparecomposerplugins)
-
-We start a docker image for php with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-In the script section we first update the environment by executing `/usr/local/bin/update-env`.
-
-After that we install the production site with `composer install` **without** the tools for development. That is pretty 
-it. We also add `drupal/core-vendor-hardening`.
-
-The entire installation is saved as a build artefact called `build-prod` where all `git` files are excluded.
-
-#### Build Prod Site NG
-
-First we have to do some preparation. This is done in the `before_script` section.
-
-- [.prepareaccess](#prepareaccess)
-- [.preparecomposerplugins](#preparecomposerplugins)
-
-We start a docker image for php with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-In the script part we install two of our LakeDrops modules:
-
-- [lakedrops/drupal-environment](../../composer/plugin/drupal-environment)
-- [lakedrops/docker4drupal](../../composer/plugin/d4d)
-
-After that we execute:
-
-- `composer lakedrops:config`
-- `composer lakedrops:docker4drupal`
-
-The entire installation is saved as a build artefact called `build-prod-ng` where all `git` files are excluded.
-
-Rules, when the job should run:
-
-- `$DISABLE_DEPLOYMENT_LOCAL` != "1"
-
-### Build
-
-Here, we define jobs, which build the Drupal website and does some additional checks. This stage is
-responsible for the database downloads as well.
-
-#### Build Site
-
-We start a docker image for `php` with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-After that, we start to build the Drupal site with [Composer](https://getcomposer.org/):
-
-Now we execute a `composer install` twice to ensure that all patches get applied correctly. After
-that we setup our own `LakeDrops` tools by `composer lakedrops:scaffold` and
-`composer lakedrops:docker4drupal`.
-
-The next step is to pull all required [Docker](https://www.docker.com/) images 
-with [Docker Compose](https://docs.docker.com/compose/) and start the services.
-
-After everything has started, we create some directories, e.g. for styles, logs and JavaScript.
-
-The entire installation is saved as a build artefact called `build` where all `git` files are excluded.
-
-Rules, when the job should run:
-
-- the tests are enabled by `$TESTSDISABLED` == "no"
-
-#### Build Theme
-
-This job creates the required files for the styling of the website. 
-
-The following jobs must run before successfully:
-
-- [Build Prod Site](#build-prod-site)
-
-#### Check 4 Updates
-
-e start a docker image for `php` with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-First we have to do some preparation. This is done in the `before_script` section.
-
-- [.prepareaccess](#prepareaccess)
-- [.preparecomposerplugins](#preparecomposerplugins)
-
-The job stores the artefacts in `check4updates`.
-
-
-
-Rules, when the job should run:
-
-- `$CHECKUPDATES` is set
-- we are on the `develop` branch
-
-
-#### Check 4 Outdated Packages
-
-We start a docker image for `php` with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
-to complete:
-
-- [Check 4 Updates](#check-4-updates)
-    - uses its artefacts
-    - optional
-- [Build Prod Site](#build-prod-site)
-    - uses its artefacts
-    - optional
-- [Build Prod Site NG](#build-prod-site-ng)
-    - uses its artefacts
-    - optional
-- [Build Site](#build-site)
-    - uses its artefacts
-    - optional
-
-First we have to do some preparation. This is done in the `before_script` section.
-
-- [.prepareaccess](#prepareaccess)
-- [.preparecomposerplugins](#preparecomposerplugins)
-
-First we perform a `composer update` which all dependencies without the development part.
-
-Then we check for outdated packages:
-
-```yml
-composer outdated --minor-only --strict --no-interaction $OUTDATED_EXTRAS
-composer outdated --patch-only --strict --no-interaction $OUTDATED_EXTRAS
-```
-
-The variable `$OUTDATED_EXTRAS` contains the ignored package like:
-`--ignore=drupal/somemodule --ignore=drupal/another`
-
-Rules, when the job must not run:
-
-- the commit message contains `SKIP_check4outdated` and `$CHECKUPDATES` != "yes"
-- the commit message contains "Merge branch 'develop' into 'main'"
-- the commit message contains "Merge branch 'release' into 'main'"
-
-Rules, when the job should run:
-
-- `$IGNORE_COMPOSER_EXTENDED_AUDIT` == "0"
-
-#### Check 4 Security Vulnerability Advisories
-
-We start a docker image for `php` with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
-to complete:
-
-- [Check 4 Updates](#check-4-updates)
-    - uses its artefacts
-    - optional
-- [Build Prod Site](#build-prod-site)
-    - uses its artefacts
-    - optional
-- [Build Prod Site NG](#build-prod-site-ng)
-    - uses its artefacts
-    - optional
-- [Build Site](#build-site)
-    - uses its artefacts
-    - optional
-
-First we have to do some preparation. This is done in the `before_script` section.
-
-- [.prepareaccess](#prepareaccess)
-- [.preparecomposerplugins](#preparecomposerplugins)
-
-First we perform a `composer update` which all dependencies without the development part.
-
-Then we check for outdated packages:
-
-```yml
-composer diagnose --no-interaction || true
-composer audit --no-dev --locked --format=table
-```
-
-Rules, when the job must not run:
-
-- the commit message contains `SKIP_check4security` and `$CHECKUPDATES` != "yes"
-- the commit message contains "Merge branch 'develop' into 'main'"
-- the commit message contains "Merge branch 'release' into 'main'"
-
-Rules, when the job should run:
-
-- `$IGNORE_COMPOSER_AUDIT` == "0"
-
-#### Download DB
-
-This job executes a script, that removes the SQL file `$]PROJECT_NAME}.sql`, if it exists already.
-After that an Ansible script is executed, which dumbs the current Drupal database of the live stage.
-
-The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`.
-
-Rules, when the job must not run:
-
-- `$DISABLE_DEPLOYMENT` != "0"
-- `$DISABLE_DEPLOYMENT_LOCAL` != "1"
-- `$DBREQUIRED` == "no"
-- `$INITIALINSTALL` == "yes"
-- `$TESTSDISABLED` == "yes"
-- `$LOCALDBFILE` != "none"
-
-Rules, when the job should run:
-
-- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
-
-#### Download DB NG
-
-This is very similar to [Download DB](#download-db), but does not use Ansible. 
-It uses Docker Compose instead to execute the `drush sql:dump` command
-to dumb the current live database to the file `$]PROJECT_NAME}.sql`.
-
-The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`.
-
-Rules, when the job must not run:
-
-- `$DISABLE_DEPLOYMENT` != "0"
-- `$DISABLE_DEPLOYMENT_LOCAL` != "0"
-- `$DBREQUIRED` == "no"
-- `$INITIALINSTALL` == "yes"
-- `$TESTSDISABLED` == "yes"
-- `$LOCALDBFILE` != "none"
-
-Rules, when the job should run:
-
-- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
-
-### Prepare
-
-This stage is used to prepare the system for testing. This stage is also used for internal purpose
-and for proprietary tools.
-
-#### Import DB
-
-This job extends `.importdb`, which itself defines the 
-anchor [import_db_default](#import-db-default-anchor).
-
-The following jobs must run before successfully:
-
-- [Build Site](#build-site)
-- [Download DB](#download-db)
-
-Rules, when the job must not run:
-
- - `$DISABLE_DEPLOYMENT` != "0"
- - `$DISABLE_DEPLOYMENT_LOCAL` != "1"
- - `$DBREQUIRED` == "no"
- - `$INITIALINSTALL` == "yes"
- - `$TESTSDISABLED` == "yes"
- - `$LOCALDBFILE` != "none"
-
-Rules, when the job should run:
-
-- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
-
-##### Import DB Default Anchor
-
-We start a docker image for `php` with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-A script copies the available SQL file into the PHP container, drops the current DB and imports the file.
-If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. After that the
-`UPDATE_DB_COMMAND` gets called as well as a cache reset.
-
-#### Import DB NG
-
-This job extends `.importdbNG`, which itself defines the
-anchor [import_db_default](#import-db-default-anchor).
-
-The following jobs must run before successfully:
-
-- [Build Site](#build-site)
-- [Download DB NG](#download-db-ng)
-
-Rules, when the job must not run:
-
-- `$DISABLE_DEPLOYMENT` != "0"
-- `$DISABLE_DEPLOYMENT_LOCAL` != "0"
-- `$DBREQUIRED` == "no"
-- `$INITIALINSTALL` == "yes"
-- `$TESTSDISABLED` == "yes"
-- `$LOCALDBFILE` != "none"
-
-Rules, when the job should run:
-
-- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
-
-#### Import Local DB
-
-This job extends `.importdbLocal`, which itself defines the
-anchor [import_db_default](#import-db-default-anchor).
-
-The following jobs must run before successfully:
-
-- [Build Site](#build-site)
-
-A before script is executed, which copies the `${LOCALDBFILE}` to `${PROJECT_NAME}.sql.gz` and
-unzips that file.
-
-Rules, when the job must not run:
-
-- `$DBREQUIRED` == "no"
-- `$TESTSDISABLED` == "yes"
-- `$LOCALDBFILE` != "none"
-
-Rules, when the job should run:
-
-- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
-
-#### Update DB
-
-We start a docker image for php with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. 
-After that the `UPDATE_DB_COMMAND` gets called as well as a cache reset.
-
-The following jobs must run before successfully:
-
-- [Build Site](#build-site)
-
-Rules, when the job must not run:
-
-- `$DBREQUIRED` == "yes"
-- `$DBUPDREQUIRED` == "no"
-- `$INITIALINSTALL` == "yes"
-- `$TESTSDISABLED` == "yes"
-
-Rules, when the job should run:
-
-- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
-
-### Postprocess
-
-#### Check 4 Updates Commit
-
-We start a docker image for php with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
-to complete:
-
-- [Check 4 Updates](#check-4-updates)
-    - uses its artefacts
-- [Check 4 Security Vulnerability Advisories](#check-4-security-vulnerability-advisories)
-    - optional
-- [Check 4 Outdated Packages](#check-4-outdated-packages)
-    - optional
-
-First we have to do some preparation. This is done in the `before_script` section.
-
-- [.prepareaccess](#prepareaccess)
-- [.preparecomposerplugins](#preparecomposerplugins)
-
-
-
-Rules, when the job should run:
-
-- `$CHECKUPDATES` is set
-- we are on the `develop` branch
-
-### Test
-
-After the preparation and the setup of a Drupal database, we can execute tests. This happens in this stage.
-
-All tests use the two [Internal Test Stages](#internal-test-stages).
-
-#### Test Code Style
-
-This job uses [PHP Code Sniffer](../../dev_tools/test.md#php-code-sniffer) and stores the artefacts in `phpcs`.
-
-Rules, when the job should run:
-
-- the code style tests are enabled by `$DISABLE_CI_TEST_CODESTYLE` != "1"
-
-#### Test PHPUnit
-
-This job uses [PHP Unit](../../dev_tools/test.md#unitkernelfunctional-testing) and stores the artefacts in `phpunit`.
-
-Rules, when the job should run:
-
-- the code style tests are enabled by `$DISABLE_CI_TEST_PHPUNIT` != "1"
-
-
-#### Test Backstop
-
-This job uses [Backstop](../../dev_tools/test.md#unitkernelfunctional-testing) and stores the artefacts in `backstop`.
-
-The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
-to complete:
-
-- [Build Site](#build-site)
-- [Build Theme](#build-theme)
-    - uses its artefacts
-    - optional
-- [Import DB](#import-db)
-    - optional 
-- [Import DB NG](#import-db-ng)
-    - optional
-- [Import Local DB](#import-local-db)
-    - optional
-- [Update DB](#update-db)
-    - optional
-
-Rules, when the job should run:
-
-- the backstop tests are enabled by `$DISABLE_CI_TEST_BACKSTOP` != "1"
-
-
-#### Test Cypress E2E
-
-This job uses [Cypress](../../dev_tools/test.md#cypress) and stores the artefacts in `cypresse2e`. The tests
-are executed in headless mode.
-
-The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
-to complete:
-
-- [Build Site](#build-site)
-- [Build Theme](#build-theme)
-    - uses its artefacts
-    - optional
-- [Import DB](#import-db)
-    - optional
-- [Import DB NG](#import-db-ng)
-    - optional
-- [Import Local DB](#import-local-db)
-    - optional
-- [Update DB](#update-db)
-    - optional
-- [Test Backstop](#test-backstop)
-    - optional
-
-Rules, when the job should run:
-
-- the cypress tests are enabled by `$DISABLE_CI_TEST_CYPRESSE2E` != "1"
-
-#### Internal Test Stages
-
-Every test stage needs internal reoccurring definitions.
-
-**.test_lakedrops_default**
-
-We start a docker image for `php` with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-The following jobs must run before successfully:
-
-- [Build Site](#build-site)
-
-But this job can start running when the [Build Site](#build-site) has finished, so other
-test jobs can run in parallel.
-
-After running, we copy the logs from the `php` container to the current directory.
-
-
-**.test_lakedrops_theme**
-
-The following jobs must run before successfully:
-
-- [Build Site](#build-site)
-- [Build Theme](#build-theme)
-
-In the `before_script` part, we copy the `$THEME_CSS_PATH` into the `php` container, if the
-`$THEME_BUILD` variable ist set to "yes".
-
-In `after_script` we remove `$THEME_CSS_PATH` again.
-
-####
-
-### Deploy
-
-This stage handles the deployment of the produced build artefact.
-
-#### Deploy
-
-The following jobs must run before successfully:
-
-- [Build Prod Site](#build-prod-site)
-- [Build Theme](#build-theme)
-
-This job executes an [Ansible](https://www.ansible.com/) script, which does the deployment.
-
-Rules, when the job must not run:
-
-- `$DISABLE_DEPLOYMENT` != "0"
-- `$DISABLE_DEPLOYMENT_LOCAL` != "1"
-
-#### Deploy NG
-
-The following jobs must run before successfully:
-
-- [Build Prod Site](#build-prod-site)
-- [Build Prod Site NG](#build-prod-site-ng)
-- [Build Theme](#build-theme)
-
-First we have to do some preparation. This is done in the `before_script` section.
-
-- [.prepareaccess](#prepareaccess)
-
-This job does not use [Ansible](https://www.ansible.com/). A lot of commands are executed in the script section.
-Here is a summary:
-
-- create the directories, where the site gets deployed to:
-    - app
-    - db
-    - files
-    - redis
-    - backup
-- sets the ownership and permissions of these files
-- start all the docker service with [Docker Compose](https://docs.docker.com/compose/)
-- execute several `drush` commands
-- create config auto export directory
-- restart cron jobs
-
-To see all the commands, visit
-[test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).
-
-Rules, when the job must not run:
-
-- `$DISABLE_DEPLOYMENT` != "0"
-- `$CI_COMMIT_BRANCH` != "main" && `$ENFORCE_DEPLOYMENT` != "1"
-
-### Finalize
-
-Here we can do cleanups and remove some garbage the build process produces.
-
-#### Shut Down Docker Project
-
-We start a docker image for php with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-The following jobs must run before successfully:
-
-- [Build Site](#build-site)
-
-This job simply stops and removes the docker container, which are started for the build by:
-
-`docker compose down`
-
-### Release Stage
-
-This stage is responsible for creating a new release.
-
-#### Release Job
-
-This job extends the internal job [.merge](#merge).
-
-Rules, when the job should run:
-
-- `$CI_COMMIT_MESSAGE` contains `CREATE_RELEASE`
-- we are on the `develop` branch
-
-Rules, when the job must not run:
-
-- if the reference is a `tag`
-- `$CI_COMMIT_MESSAGE` contains `Merge tag`
-- if it is triggered by config auto export (CAE)
-- `$CHECKUPDATES` is set
-- `$TRIGGERTASK` is set
-
-
-### Internal Stages
-
-Here we have some internal stages, which cannot be executed alone. The purpose is to
-collect repeating execution.
-
-#### .prepareaccess
-
-This internal job, you find in [mixins.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/general/-/blob/main/mixins.yml?ref_type=heads).
-
-Mainly, we care about getting access to:
-
-- the host `$CI_SERVER_HOST` with `ssh` via `$SSH_PRIVATE_KEY`
-- GitLab Login via `$GITLAB_ACCESS_TOKEN` and `$GITLAB_PRIVATE_TOKEN`
-- Docker configuration via `$DOCKER_AUTH_CONFIG`
-
-
-#### .preparecomposerplugins
-
-This internal job, you find in [mixins.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/general/-/blob/main/mixins.yml?ref_type=heads).
-
-The following points will be prepared for [Composer](https://getcomposer.org/):
-
-- the version >2, if `$DOWNGRADE_COMPOSER` is not set
-- the configuration `allow-plugins`
-- the configuration `audit.abandoned`, depend on `$IGNORE_COMPOSER_ABANDONED_AUDIT`
-- unsupported module versions, which are allowed to use
-
-#### .merge
-
-First we have to do some preparation. This is done in the `before_script` section.
-
-- [.prepareaccess](#prepareaccess)
-
-We start a docker image for php with the specifies versions:
-
-- `PHP_MAJOR_VERSION`
-- `PHP_MINOR_VERSION`
-
-We create a clean directory and clone the git project `$CI_PROJECT_PATH`
-for branch `$SOURCE_BRANCH` from `$CI_SERVER_HOST` right into it.
-
-After that we execute the [merge](https://gitlab.lakedrops.com/docker/gitlab-drupal-ci/-/blob/main/bin/merge?ref_type=heads) 
-script, which does a few checks and use the `gitlab` tool to merge.
-
-Finally, we remove the directory we created above.
-
-#### .retry-for-system-issues
-
-This little job ste the amount of retries to "2", when:
-
-- runner_system_failure
-- stuck_or_timeout_failure
-
-#### .cache_paths
-
-All internal jobs concerning cache will use this job here. It defines the paths:
-
-- .docker-init/
-- assets/
-- drush/
-- files/
-- keys/
-- settings/
-- vendor/
-- web/core/
-- web/libraries/
-- web/modules/contrib/
-- web/profiles/contrib/
-- web/sites/
-- web/themes/contrib/
-- .ahoy.yml
-- .env
-- docker-compose.yml
-
-
-## Configuration
-
-### GitLab Runner
+## GitLab Runner
 
 To setup a GitLab runner, they need to be
 [installed first](https://docs.gitlab.com/runner/install) - see also our
@@ -813,7 +47,7 @@ different PHP versions a different GitLab runner should be configured or the
 image being overwritten for each task in the `.gitlab-ci.yml` of the respective
 project.
 
-### GitLab Variables
+## GitLab Variables
 
 Some variables are required for proper access control, and you have to provide
 them in the project configuration by going to **Settings / CI/CD / Variables**
@@ -825,7 +59,7 @@ and adding these variables:
   repository, you have to create an access tokeb for the user who runs the
   pipeline and provide it in this variable.
 
-### GitLab CI instructions in Drupal project
+## GitLab CI instructions in Drupal project
 
 The Drupal project should be setup with the
 [Drupal Development Environment](/composer/plugin/drupal-dev-environment)
@@ -930,7 +164,7 @@ using our Ansible environment too, then please refer to
 [Using Ansible for Deployment](#Using Ansible for Deployment) below. Otherwise,
 you have to write the scripts for those two tasks yourselves.
 
-#### Variables
+### Variables
 
 - `COMPOSE_PROJECT_NAME`: a string only unique project name which will be used
   to identify caches and Docker containers.
@@ -969,11 +203,11 @@ you have to write the scripts for those two tasks yourselves.
   completely and built from scratch.  Can also be used as
   `[RESET_LOCALE]` in the Git commit message. 
 
-## Usage
+# Usage
 
 This chapter is incomplete so far and needs more attention.
 
-### Initial installation of a Drupal site
+## Initial installation of a Drupal site
 
 This pipeline covers both, the initial installation and later updates of a
 Drupal site. The latter is the default, because it happens regularely whereas
@@ -987,7 +221,7 @@ pipeline from the GitLab UI.
 When either of these conditions apply, no database will be downloaded, imported
 or updated and all tests will be skipped.
 
-### Handling of the database
+## Handling of the database
 
 To build and test your Drupal site prior to deployment, a database with content
 for this project is required and this is handled by the pipeline pretty smart.
@@ -1012,7 +246,7 @@ task `Import DB` will then import that dump in the `prepare` stage.
 Otherwise, those two tasks will be skipped and the task `Update DB` will be
 executed in the `prepare` stage instead.
 
-## Example
+# Example
 
 We provide a [Demo Drupal 10 project](https://gitlab.lakedrops.com/gitlab-ci-cd/demos/drupal-10)
 which uses this GitLab CI framework. If you want to give it a try, login to 

docs/gitlab/pipelines/index.md

+---
+title: GitLab Drupal
+repo: https://gitlab.lakedrops.com/gitlab-ci-cd/pipelines
+issues: https://gitlab.lakedrops.com/gitlab-ci-cd/pipelines/-/issues
+tags:
+- gitlab
+- ci/cd
+- pipeline
+- job
+---
+# GitLab CI for Drupal pipelines
+
+This project contains a number of pre-configured tasks for GitLab CI/CD that
+allow to run very powerful pipelines which contain the following stage order:
+
+- [.pre (predefined)](#pre)
+- [execute](#execute)
+- [buildprod](#build-prod)
+- [build (predefined)](#build)
+- [prepare](#prepare)
+- [postprocess](#postprocess)
+- [validate](#validate)
+- [test (predefined)](#test)
+- [deploy (predefined)](#deploy)
+- [finalize](#finalize)
+- [release](#release-stage)
+
+We have a predefined `execute` job called `LakeDrops`, which looks like this:
+
+- [execute](#execute)
+    - LakeDrops
+        - [buildprod](#build-prod)
+            - [Build Prod Site](#build-prod-site)
+            - [Build Prod Site NG](#build-prod-site-ng)
+        - [build (predefined)](#build)
+            - [Build Site](#build-site)
+            - [Build Theme](#build-theme)
+            - [Check 4 Outdated Packages](#check-4-outdated-packages)
+            - [Check 4 Security Vulnerability Advisories](#check-4-security-vulnerability-advisories)
+            - [Download DB](#download-db)
+            - [Download DB NG](#download-db-ng)
+        - [prepare](#prepare)
+            - [Import DB](#import-db)
+            - [Import DB NG](#import-db-ng)
+            - [Import Local DB](#import-local-db)
+            - [Update DB](#update-db) 
+        - [test](#test)
+            - [Test Code Style](#test-code-style)
+            - [Test PHPUnit](#test-phpunit)
+            - [Test Backstop](#test-backstop)
+            - [Test Cypress E2E](#test-cypress-e2e)
+        - [finalize](#finalize)
+            - [Shut Down Docker Project](#shut-down-docker-project)
+
+These pre-configured tasks can easily be included into your Drupal project.
+A GitLab runner on any host can then be configured to run those pipelines. The 
+following chapters describe the necessary steps and also all the options
+available for configuration.
+
+## Stages
+The following chapters describes all stages and their jobs. The files:
+
+-  [lakedrops.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/lakedrops.yml?ref_type=heads).
+-  [tests/lakedrops.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/tests/lakedrops.yml?ref_type=heads).
+-  [test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).
+-  [check4updates.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/check4updates.yml?ref_type=heads).
+
+### .pre
+
+To prepare and check, if the environment is good to go, we have a `.pre` stage, which contains the
+following jobs:
+
+#### Sanity Checks
+
+The sanity check gets important when you are using [Crowdsec](https://www.crowdsec.net/).
+It contains two little scripts, that check:
+
+-    if the `crowdsec.settings.yml` is in the config split and
+-    if it is not enabled in the `core.extensions.yml`.
+
+Otherwise, it is conflicting with you local settings, since `CrowdSec`
+is a tool for production sites.
+
+Except:
+
+- Commit message contains "Merge tag"
+- Commit message contains "Merge branch"
+- When the job is triggered by config auto export (CAE)
+
+#### Validate Environment
+
+This job extends the `.prerequisites` in 
+[test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).
+
+This job executes a script, where several [variables](#variables) are checked and set accordingly.
+
+Except:
+
+- Commit message contains "Merge tag"
+- Commit message contains "Merge branch"
+- When the job is triggered by config auto export (CAE)
+
+### Validate
+
+This stage is currently not officially used.
+
+### Execute
+
+This stage executes the build itself.
+
+### Build Prod
+
+This stage is responsible for building the production website, which we want to finally deploy.
+
+#### Build Prod Site
+
+First we have to do some preparation. This is done in the `before_script` section.
+
+- [.prepareaccess](#prepareaccess)
+- [.preparecomposerplugins](#preparecomposerplugins)
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+In the script section we first update the environment by executing `/usr/local/bin/update-env`.
+
+After that we install the production site with `composer install` **without** the tools for development. That is pretty 
+it. We also add `drupal/core-vendor-hardening`.
+
+The entire installation is saved as a build artefact called `build-prod` where all `git` files are excluded.
+
+#### Build Prod Site NG
+
+First we have to do some preparation. This is done in the `before_script` section.
+
+- [.prepareaccess](#prepareaccess)
+- [.preparecomposerplugins](#preparecomposerplugins)
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+In the script part we install two of our LakeDrops modules:
+
+- [lakedrops/drupal-environment](../../composer/plugin/drupal-environment)
+- [lakedrops/docker4drupal](../../composer/plugin/d4d)
+
+After that we execute:
+
+- `composer lakedrops:config`
+- `composer lakedrops:docker4drupal`
+
+The entire installation is saved as a build artefact called `build-prod-ng` where all `git` files are excluded.
+
+Rules, when the job should run:
+
+- `$DISABLE_DEPLOYMENT_LOCAL` != "1"
+
+### Build
+
+Here, we define jobs, which build the Drupal website and does some additional checks. This stage is
+responsible for the database downloads as well.
+
+#### Build Site
+
+We start a docker image for `php` with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+After that, we start to build the Drupal site with [Composer](https://getcomposer.org/):
+
+Now we execute a `composer install` twice to ensure that all patches get applied correctly. After
+that we setup our own `LakeDrops` tools by `composer lakedrops:scaffold` and
+`composer lakedrops:docker4drupal`.
+
+The next step is to pull all required [Docker](https://www.docker.com/) images 
+with [Docker Compose](https://docs.docker.com/compose/) and start the services.
+
+After everything has started, we create some directories, e.g. for styles, logs and JavaScript.
+
+The entire installation is saved as a build artefact called `build` where all `git` files are excluded.
+
+Rules, when the job should run:
+
+- the tests are enabled by `$TESTSDISABLED` == "no"
+
+#### Build Theme
+
+This job creates the required files for the styling of the website. 
+
+The following jobs must run before successfully:
+
+- [Build Prod Site](#build-prod-site)
+
+#### Check 4 Updates
+
+e start a docker image for `php` with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+First we have to do some preparation. This is done in the `before_script` section.
+
+- [.prepareaccess](#prepareaccess)
+- [.preparecomposerplugins](#preparecomposerplugins)
+
+The job stores the artefacts in `check4updates`.
+
+
+
+Rules, when the job should run:
+
+- `$CHECKUPDATES` is set
+- we are on the `develop` branch
+
+
+#### Check 4 Outdated Packages
+
+We start a docker image for `php` with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
+to complete:
+
+- [Check 4 Updates](#check-4-updates)
+    - uses its artefacts
+    - optional
+- [Build Prod Site](#build-prod-site)
+    - uses its artefacts
+    - optional
+- [Build Prod Site NG](#build-prod-site-ng)
+    - uses its artefacts
+    - optional
+- [Build Site](#build-site)
+    - uses its artefacts
+    - optional
+
+First we have to do some preparation. This is done in the `before_script` section.
+
+- [.prepareaccess](#prepareaccess)
+- [.preparecomposerplugins](#preparecomposerplugins)
+
+First we perform a `composer update` which all dependencies without the development part.
+
+Then we check for outdated packages:
+
+```yml
+composer outdated --minor-only --strict --no-interaction $OUTDATED_EXTRAS
+composer outdated --patch-only --strict --no-interaction $OUTDATED_EXTRAS
+```
+
+The variable `$OUTDATED_EXTRAS` contains the ignored package like:
+`--ignore=drupal/somemodule --ignore=drupal/another`
+
+Rules, when the job must not run:
+
+- the commit message contains `SKIP_check4outdated` and `$CHECKUPDATES` != "yes"
+- the commit message contains "Merge branch 'develop' into 'main'"
+- the commit message contains "Merge branch 'release' into 'main'"
+
+Rules, when the job should run:
+
+- `$IGNORE_COMPOSER_EXTENDED_AUDIT` == "0"
+
+#### Check 4 Security Vulnerability Advisories
+
+We start a docker image for `php` with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
+to complete:
+
+- [Check 4 Updates](#check-4-updates)
+    - uses its artefacts
+    - optional
+- [Build Prod Site](#build-prod-site)
+    - uses its artefacts
+    - optional
+- [Build Prod Site NG](#build-prod-site-ng)
+    - uses its artefacts
+    - optional
+- [Build Site](#build-site)
+    - uses its artefacts
+    - optional
+
+First we have to do some preparation. This is done in the `before_script` section.
+
+- [.prepareaccess](#prepareaccess)
+- [.preparecomposerplugins](#preparecomposerplugins)
+
+First we perform a `composer update` which all dependencies without the development part.
+
+Then we check for outdated packages:
+
+```yml
+composer diagnose --no-interaction || true
+composer audit --no-dev --locked --format=table
+```
+
+Rules, when the job must not run:
+
+- the commit message contains `SKIP_check4security` and `$CHECKUPDATES` != "yes"
+- the commit message contains "Merge branch 'develop' into 'main'"
+- the commit message contains "Merge branch 'release' into 'main'"
+
+Rules, when the job should run:
+
+- `$IGNORE_COMPOSER_AUDIT` == "0"
+
+#### Download DB
+
+This job executes a script, that removes the SQL file `$]PROJECT_NAME}.sql`, if it exists already.
+After that an Ansible script is executed, which dumbs the current Drupal database of the live stage.
+
+The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`.
+
+Rules, when the job must not run:
+
+- `$DISABLE_DEPLOYMENT` != "0"
+- `$DISABLE_DEPLOYMENT_LOCAL` != "1"
+- `$DBREQUIRED` == "no"
+- `$INITIALINSTALL` == "yes"
+- `$TESTSDISABLED` == "yes"
+- `$LOCALDBFILE` != "none"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+#### Download DB NG
+
+This is very similar to [Download DB](#download-db), but does not use Ansible. 
+It uses Docker Compose instead to execute the `drush sql:dump` command
+to dumb the current live database to the file `$]PROJECT_NAME}.sql`.
+
+The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`.
+
+Rules, when the job must not run:
+
+- `$DISABLE_DEPLOYMENT` != "0"
+- `$DISABLE_DEPLOYMENT_LOCAL` != "0"
+- `$DBREQUIRED` == "no"
+- `$INITIALINSTALL` == "yes"
+- `$TESTSDISABLED` == "yes"
+- `$LOCALDBFILE` != "none"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+### Prepare
+
+This stage is used to prepare the system for testing. This stage is also used for internal purpose
+and for proprietary tools.
+
+#### Import DB
+
+This job extends `.importdb`, which itself defines the 
+anchor [import_db_default](#import-db-default-anchor).
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+- [Download DB](#download-db)
+
+Rules, when the job must not run:
+
+ - `$DISABLE_DEPLOYMENT` != "0"
+ - `$DISABLE_DEPLOYMENT_LOCAL` != "1"
+ - `$DBREQUIRED` == "no"
+ - `$INITIALINSTALL` == "yes"
+ - `$TESTSDISABLED` == "yes"
+ - `$LOCALDBFILE` != "none"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+##### Import DB Default Anchor
+
+We start a docker image for `php` with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+A script copies the available SQL file into the PHP container, drops the current DB and imports the file.
+If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. After that the
+`UPDATE_DB_COMMAND` gets called as well as a cache reset.
+
+#### Import DB NG
+
+This job extends `.importdbNG`, which itself defines the
+anchor [import_db_default](#import-db-default-anchor).
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+- [Download DB NG](#download-db-ng)
+
+Rules, when the job must not run:
+
+- `$DISABLE_DEPLOYMENT` != "0"
+- `$DISABLE_DEPLOYMENT_LOCAL` != "0"
+- `$DBREQUIRED` == "no"
+- `$INITIALINSTALL` == "yes"
+- `$TESTSDISABLED` == "yes"
+- `$LOCALDBFILE` != "none"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+#### Import Local DB
+
+This job extends `.importdbLocal`, which itself defines the
+anchor [import_db_default](#import-db-default-anchor).
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+
+A before script is executed, which copies the `${LOCALDBFILE}` to `${PROJECT_NAME}.sql.gz` and
+unzips that file.
+
+Rules, when the job must not run:
+
+- `$DBREQUIRED` == "no"
+- `$TESTSDISABLED` == "yes"
+- `$LOCALDBFILE` != "none"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+#### Update DB
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. 
+After that the `UPDATE_DB_COMMAND` gets called as well as a cache reset.
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+
+Rules, when the job must not run:
+
+- `$DBREQUIRED` == "yes"
+- `$DBUPDREQUIRED` == "no"
+- `$INITIALINSTALL` == "yes"
+- `$TESTSDISABLED` == "yes"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+### Postprocess
+
+#### Check 4 Updates Commit
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
+to complete:
+
+- [Check 4 Updates](#check-4-updates)
+    - uses its artefacts
+- [Check 4 Security Vulnerability Advisories](#check-4-security-vulnerability-advisories)
+    - optional
+- [Check 4 Outdated Packages](#check-4-outdated-packages)
+    - optional
+
+First we have to do some preparation. This is done in the `before_script` section.
+
+- [.prepareaccess](#prepareaccess)
+- [.preparecomposerplugins](#preparecomposerplugins)
+
+
+
+Rules, when the job should run:
+
+- `$CHECKUPDATES` is set
+- we are on the `develop` branch
+
+### Test
+
+After the preparation and the setup of a Drupal database, we can execute tests. This happens in this stage.
+
+All tests use the two [Internal Test Stages](#internal-test-stages).
+
+#### Test Code Style
+
+This job uses [PHP Code Sniffer](../../dev_tools/test.md#php-code-sniffer) and stores the artefacts in `phpcs`.
+
+Rules, when the job should run:
+
+- the code style tests are enabled by `$DISABLE_CI_TEST_CODESTYLE` != "1"
+
+#### Test PHPUnit
+
+This job uses [PHP Unit](../../dev_tools/test.md#unitkernelfunctional-testing) and stores the artefacts in `phpunit`.
+
+Rules, when the job should run:
+
+- the code style tests are enabled by `$DISABLE_CI_TEST_PHPUNIT` != "1"
+
+
+#### Test Backstop
+
+This job uses [Backstop](../../dev_tools/test.md#unitkernelfunctional-testing) and stores the artefacts in `backstop`.
+
+The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
+to complete:
+
+- [Build Site](#build-site)
+- [Build Theme](#build-theme)
+    - uses its artefacts
+    - optional
+- [Import DB](#import-db)
+    - optional 
+- [Import DB NG](#import-db-ng)
+    - optional
+- [Import Local DB](#import-local-db)
+    - optional
+- [Update DB](#update-db)
+    - optional
+
+Rules, when the job should run:
+
+- the backstop tests are enabled by `$DISABLE_CI_TEST_BACKSTOP` != "1"
+
+
+#### Test Cypress E2E
+
+This job uses [Cypress](../../dev_tools/test.md#cypress) and stores the artefacts in `cypresse2e`. The tests
+are executed in headless mode.
+
+The following jobs must run before successfully. Some are optional, which means, if they exist, this job waits for them
+to complete:
+
+- [Build Site](#build-site)
+- [Build Theme](#build-theme)
+    - uses its artefacts
+    - optional
+- [Import DB](#import-db)
+    - optional
+- [Import DB NG](#import-db-ng)
+    - optional
+- [Import Local DB](#import-local-db)
+    - optional
+- [Update DB](#update-db)
+    - optional
+- [Test Backstop](#test-backstop)
+    - optional
+
+Rules, when the job should run:
+
+- the cypress tests are enabled by `$DISABLE_CI_TEST_CYPRESSE2E` != "1"
+
+#### Internal Test Stages
+
+Every test stage needs internal reoccurring definitions.
+
+**.test_lakedrops_default**
+
+We start a docker image for `php` with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+
+But this job can start running when the [Build Site](#build-site) has finished, so other
+test jobs can run in parallel.
+
+After running, we copy the logs from the `php` container to the current directory.
+
+
+**.test_lakedrops_theme**
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+- [Build Theme](#build-theme)
+
+In the `before_script` part, we copy the `$THEME_CSS_PATH` into the `php` container, if the
+`$THEME_BUILD` variable ist set to "yes".
+
+In `after_script` we remove `$THEME_CSS_PATH` again.
+
+####
+
+### Deploy
+
+This stage handles the deployment of the produced build artefact.
+
+#### Deploy
+
+The following jobs must run before successfully:
+
+- [Build Prod Site](#build-prod-site)
+- [Build Theme](#build-theme)
+
+This job executes an [Ansible](https://www.ansible.com/) script, which does the deployment.
+
+Rules, when the job must not run:
+
+- `$DISABLE_DEPLOYMENT` != "0"
+- `$DISABLE_DEPLOYMENT_LOCAL` != "1"
+
+#### Deploy NG
+
+The following jobs must run before successfully:
+
+- [Build Prod Site](#build-prod-site)
+- [Build Prod Site NG](#build-prod-site-ng)
+- [Build Theme](#build-theme)
+
+First we have to do some preparation. This is done in the `before_script` section.
+
+- [.prepareaccess](#prepareaccess)
+
+This job does not use [Ansible](https://www.ansible.com/). A lot of commands are executed in the script section.
+Here is a summary:
+
+- create the directories, where the site gets deployed to:
+    - app
+    - db
+    - files
+    - redis
+    - backup
+- sets the ownership and permissions of these files
+- start all the docker service with [Docker Compose](https://docs.docker.com/compose/)
+- execute several `drush` commands
+- create config auto export directory
+- restart cron jobs
+
+To see all the commands, visit
+[test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).
+
+Rules, when the job must not run:
+
+- `$DISABLE_DEPLOYMENT` != "0"
+- `$CI_COMMIT_BRANCH` != "main" && `$ENFORCE_DEPLOYMENT` != "1"
+
+### Finalize
+
+Here we can do cleanups and remove some garbage the build process produces.
+
+#### Shut Down Docker Project
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+
+This job simply stops and removes the docker container, which are started for the build by:
+
+`docker compose down`
+
+### Release Stage
+
+This stage is responsible for creating a new release.
+
+#### Release Job
+
+This job extends the internal job [.merge](#merge).
+
+Rules, when the job should run:
+
+- `$CI_COMMIT_MESSAGE` contains `CREATE_RELEASE`
+- we are on the `develop` branch
+
+Rules, when the job must not run:
+
+- if the reference is a `tag`
+- `$CI_COMMIT_MESSAGE` contains `Merge tag`
+- if it is triggered by config auto export (CAE)
+- `$CHECKUPDATES` is set
+- `$TRIGGERTASK` is set
+
+
+### Internal Stages
+
+Here we have some internal stages, which cannot be executed alone. The purpose is to
+collect repeating execution.
+
+#### .prepareaccess
+
+This internal job, you find in [mixins.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/general/-/blob/main/mixins.yml?ref_type=heads).
+
+Mainly, we care about getting access to:
+
+- the host `$CI_SERVER_HOST` with `ssh` via `$SSH_PRIVATE_KEY`
+- GitLab Login via `$GITLAB_ACCESS_TOKEN` and `$GITLAB_PRIVATE_TOKEN`
+- Docker configuration via `$DOCKER_AUTH_CONFIG`
+
+
+#### .preparecomposerplugins
+
+This internal job, you find in [mixins.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/general/-/blob/main/mixins.yml?ref_type=heads).
+
+The following points will be prepared for [Composer](https://getcomposer.org/):
+
+- the version >2, if `$DOWNGRADE_COMPOSER` is not set
+- the configuration `allow-plugins`
+- the configuration `audit.abandoned`, depend on `$IGNORE_COMPOSER_ABANDONED_AUDIT`
+- unsupported module versions, which are allowed to use
+
+#### .merge
+
+First we have to do some preparation. This is done in the `before_script` section.
+
+- [.prepareaccess](#prepareaccess)
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+We create a clean directory and clone the git project `$CI_PROJECT_PATH`
+for branch `$SOURCE_BRANCH` from `$CI_SERVER_HOST` right into it.
+
+After that we execute the [merge](https://gitlab.lakedrops.com/docker/gitlab-drupal-ci/-/blob/main/bin/merge?ref_type=heads) 
+script, which does a few checks and use the `gitlab` tool to merge.
+
+Finally, we remove the directory we created above.
+
+#### .retry-for-system-issues
+
+This little job ste the amount of retries to "2", when:
+
+- runner_system_failure
+- stuck_or_timeout_failure
+
+#### .cache_paths
+
+All internal jobs concerning cache will use this job here. It defines the paths:
+
+- .docker-init/
+- assets/
+- drush/
+- files/
+- keys/
+- settings/
+- vendor/
+- web/core/
+- web/libraries/
+- web/modules/contrib/
+- web/profiles/contrib/
+- web/sites/
+- web/themes/contrib/
+- .ahoy.yml
+- .env
+- docker-compose.yml

mkdocs.yml

@@ -266,6 +266,7 @@ nav:
   - GitLab:
       - gitlab/index.md
       - Drupal: gitlab/drupal/index.md
+      - Pipelines: gitlab/pipelines/index.md
       - Ansible: gitlab/ansible/index.md
   - DevOps:
       - ALM: