Merge branch 'develop' into 'main'

Develop See merge request !22

Merge branch 'develop' into 'main'
4e031e65 · danielspeicher · b0542398 · 5f2a6a32 · 4e031e65 · 4e031e65
Commit 4e031e65 authored 8 months ago by danielspeicher
--- a/.gitignore
+++ b/.gitignore
@@ -14,3 +14,4 @@
 /.idea/vcs.xml
 .gitignore
 /site/
+/.idea/git_toolbox_blame.xml
--- a/docs/gitlab/drupal/index.md
+++ b/docs/gitlab/drupal/index.md
@@ -12,18 +12,373 @@ tags:
 This project contains a number of pre-configured tasks for GitLab CI/CD that
 allow to run very powerful pipelines which contain the following stages:

- validation
+- .pre (predefined)
+- validate
+- execute
+- build (predefined)
 - buildprod
- build
 - prepare
- test
- deploy
+- test (predefined)
+- deploy (predefined)
+- release
+- finalize
+- postprocess
+- .post (predefined)
+
+We have a predefined `execute` job called `LakeDrops`, which looks like this:
+
+- execute
+    - LakeDrops
+        - buildprod
+            - Build Prod Site
+            - Build Prod Site NG
+        - build
+            - Build Site
+            - Build Theme
+            - Check 4 Outdated Packages
+            - Check 4 Security Vulnerability Advisories
+            - Download DB
+            - Download DB NG
+        - prepare
+            - Import DB
+            - Import DB NG
+            - Import Local DB
+            - Update DB 
+        - test
+            - Test Code Style
+            - Test PHPUnit
+            - Test Backstop
+            - Test Cypress E2E
+        - finalize
+            - Shut Down Docker Project

 These pre-configured tasks can easily be included into your Drupal project.
 A GitLab runner on any host can then be configured to run those pipelines. The 
 following chapters describe the necessary steps and also all the options
 available for configuration.

+## Stages
+The following chapters describes all stages and their jobs. The files:
+
+-  [lakedrops.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/lakedrops.yml?ref_type=heads).
+-  [test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).
+
+### .pre
+To prepare and check, if the environment is good to go, we have a `.pre` stage, which contains the
+following jobs:
+
+#### Sanity Checks
+
+The sanity check gets important when you are using [Crowdsec](https://www.crowdsec.net/).
+It contains two little scripts, that check:
+
+-    if the `crowdsec.settings.yml` is in the config split and
+-    if it is not enabled in the `core.extensions.yml`.
+
+Otherwise, it is conflicting with you local settings, since `CrowdSec`
+is a tool for production sites.
+
+Except:
+
+- Commit message contains "Merge tag"
+- Commit message contains "Merge branch"
+- When the job is triggered by config auto export (CAE)
+
+#### Validate Environment
+
+This job extends the `.prerequisites` in 
+[test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads).
+
+This job executes a script, where several [variables](#variables) are checked and set accordingly.
+
+Except:
+
+- Commit message contains "Merge tag"
+- Commit message contains "Merge branch"
+- When the job is triggered by config auto export (CAE)
+
+### Validate
+
+This stage is currently not officially used.
+
+### Execute
+
+This stage executes the build itself.
+
+### Build Prod
+
+This stage is responsible for building the production website, which we want to finally deploy.
+
+#### Build Prod Site
+
+First we have to do some preparation. This is done in the `before_script` section.
+
+- [.prepareaccess](#prepareaccess)
+- [.preparecomposerplugins](#preparecomposerplugins)
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+In the script section we first update the environment by executing `/usr/local/bin/update-env`.
+
+After that we install the production site with `composer install` **without** the tools for development. That is pretty 
+it. We also add `drupal/core-vendor-hardening`.
+
+The entire installation is saved as a build artefact called `build-prod` where all `git` files are excluded.
+
+#### Build Prod Site NG
+
+First we have to do some preparation. This is done in the `before_script` section.
+
+- [.prepareaccess](#prepareaccess)
+- [.preparecomposerplugins](#preparecomposerplugins)
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+In the script part we install two of our LakeDrops modules:
+
+- [lakedrops/drupal-environment](../../composer/plugin/drupal-environment)
+- [lakedrops/docker4drupal](../../composer/plugin/d4d)
+
+After that we execute:
+
+- `composer lakedrops:config`
+- `composer lakedrops:docker4drupal`
+
+The entire installation is saved as a build artefact called `build-prod-ng` where all `git` files are excluded.
+
+Rules, when the job should run:
+
+- `$DISABLE_DEPLOYMENT_LOCAL` != "1"
+
+### Build
+
+Here, we define jobs, which build the Drupal website and does some additional checks. This stage is
+responsible for the database downloads as well.
+
+#### Build Site
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+After that, we start to build the Drupal site with [Composer](https://getcomposer.org/):
+
+Now we execute a `composer install` twice to ensure that all patches get applied correctly. After
+that we setup our own `LakeDrops` tools by `composer lakedrops:scaffold` and
+`composer lakedrops:docker4drupal`.
+
+The next step is to pull all required [Docker](https://www.docker.com/) images 
+with [Docker Compose](https://docs.docker.com/compose/) adn start the services.
+
+After everything has started, we create some directories, e.g. for styles, logs and JavaScript.
+
+The entire installation is saved as a build artefact called `build` where all `git` files are excluded.
+
+Rules, when the job should run:
+
+- the tests are enabled by `$TESTSDISABLED` == "no"
+
+#### Build Theme
+
+This job creates the required files for the styling of the website. 
+
+The following jobs must run before successfully:
+
+- [Build Prod Site](#build-prod-site)
+
+
+
+#### Download DB
+
+This job executes a script, that removes the SQL file `$]PROJECT_NAME}.sql`, if it exists already.
+After that an Ansible script is executed, which dumbs the current Drupal database of the live stage.
+
+The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`.
+
+Rules, when the job must not run:
+
+- `$DISABLE_DEPLOYMENT` != "0"
+- `$DISABLE_DEPLOYMENT_LOCAL` != "1"
+- `$DBREQUIRED` == "no"
+- `$INITIALINSTALL` == "yes"
+- `$TESTSDISABLED` == "yes"
+- `$LOCALDBFILE` != "none"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+#### Download DB NG
+
+This is very similar to [Download DB](#download-db), but does not use Ansible. 
+It uses Docker Compose instead to execute the `drush sql:dump` command
+to dumb the current live database to the file `$]PROJECT_NAME}.sql`.
+
+The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`.
+
+Rules, when the job must not run:
+
+- `$DISABLE_DEPLOYMENT` != "0"
+- `$DISABLE_DEPLOYMENT_LOCAL` != "0"
+- `$DBREQUIRED` == "no"
+- `$INITIALINSTALL` == "yes"
+- `$TESTSDISABLED` == "yes"
+- `$LOCALDBFILE` != "none"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+### Prepare
+
+This stage is used to prepare the system for testing. This stage is also used for internal purpose
+and for proprietary tools.
+
+#### Import DB
+
+This job extends `.importdb`, which itself defines the 
+anchor [import_db_default](#import-db-default-anchor).
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+- [Download DB](#download-db)
+
+Rules, when the job must not run:
+
+ - `$DISABLE_DEPLOYMENT` != "0"
+ - `$DISABLE_DEPLOYMENT_LOCAL` != "1"
+ - `$DBREQUIRED` == "no"
+ - `$INITIALINSTALL` == "yes"
+ - `$TESTSDISABLED` == "yes"
+ - `$LOCALDBFILE` != "none"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+##### Import DB Default Anchor
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+A script copies the available SQL file into the PHP container, drops the current DB and imports the file.
+If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. After that the
+`UPDATE_DB_COMMAND` gets called as well as a cache reset.
+
+#### Import DB NG
+
+This job extends `.importdbNG`, which itself defines the
+anchor [import_db_default](#import-db-default-anchor).
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+- [Download DB NG](#download-db-ng)
+
+Rules, when the job must not run:
+
+- `$DISABLE_DEPLOYMENT` != "0"
+- `$DISABLE_DEPLOYMENT_LOCAL` != "0"
+- `$DBREQUIRED` == "no"
+- `$INITIALINSTALL` == "yes"
+- `$TESTSDISABLED` == "yes"
+- `$LOCALDBFILE` != "none"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+#### Import Local DB
+
+This job extends `.importdbLocal`, which itself defines the
+anchor [import_db_default](#import-db-default-anchor).
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+
+A before script is executed, which copies the `${LOCALDBFILE}` to `${PROJECT_NAME}.sql.gz` and
+unzips that file.
+
+Rules, when the job must not run:
+
+- `$DBREQUIRED` == "no"
+- `$TESTSDISABLED` == "yes"
+- `$LOCALDBFILE` != "none"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+#### Update DB
+
+We start a docker image for php with the specifies versions:
+
+- `PHP_MAJOR_VERSION`
+- `PHP_MINOR_VERSION`
+
+If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. 
+After that the `UPDATE_DB_COMMAND` gets called as well as a cache reset.
+
+The following jobs must run before successfully:
+
+- [Build Site](#build-site)
+
+Rules, when the job must not run:
+
+- `$DBREQUIRED` == "yes"
+- `$DBUPDREQUIRED` == "no"
+- `$INITIALINSTALL` == "yes"
+- `$TESTSDISABLED` == "yes"
+
+Rules, when the job should run:
+
+- the tests are enabled by `$DISABLE_CI_TESTS` != "1"
+
+### Deploy
+
+This stage handles the deployment of the produced build artefact.
+
+#### Deploy
+
+The following jobs must run before successfully:
+
+- [Build Prod Site](#build-prod-site)
+- [Build Theme](#build-theme)
+
+This job executes an [Ansible](https://www.ansible.com/) script, which does the deployment.
+
+Rules, when the job must not run:
+
+- `$DISABLE_DEPLOYMENT` != "0"
+- `$DISABLE_DEPLOYMENT_LOCAL` != "1"
+
+#### Deploy NG
+
+The following jobs must run before successfully:
+
+- [Build Prod Site](#build-prod-site)
+- [Build Prod Site NG](#build-prod-site-ng)
+- [Build Theme](#build-theme)
+
+### Internal Stages
+
+Here we have some internal stages, which cannot be executed alone. The purpose is to
+collect repeating execution.
+
+#### .prepareaccess
+#### .preparecomposerplugins
+
 ## Configuration

 ### GitLab Runner
@@ -187,7 +542,7 @@ you have to write the scripts for those two tasks yourselves.
 - `ENVIRONMENT_NAME`: the environment name for the GitLab UI.
 - `THEME_CSS_PATH`: the relative path to the theme's css path where the
  generated artefact can be found.
- `PHP_MAJOR_VERSION`: the major PHP version, defaults to 7.
+- `PHP_MAJOR_VERSION`: the major PHP version, defaults to 10.
 - `PHP_MINOR_VERSION`: the minor PHP version, default to 4.
 - `COMPOSER_DOWNGRADE`: by default, composer 2 is being used. To use composer 1
  instead, set this variable to 1.
@@ -201,6 +556,8 @@ you have to write the scripts for those two tasks yourselves.
  test stage will be skipped. This is useful e.g. in a development environment
  where you push and run pipelines often but don't want to run the tests every
  single time.
+- `$DISABLE_GITLAB_CI_TESTS`: if this variable is set to "1", no GitLab Tests will be
+  executed, see: [GitLab Templates](https://www.drupal.org/project/gitlab_templates).
 - `DISABLE_CI_TEST_BACKSTOP`: if this variable is set to any value, the visual
  regression tests with Backstop will be skipped.
 - `DISABLE_CI_TEST_BEHAT`: if this variable is set to any value, the Behat
@@ -217,30 +574,6 @@ you have to write the scripts for those two tasks yourselves.
  completely and built from scratch.  Can also be used as
  `[RESET_LOCALE]` in the Git commit message. 

-#### Stages and Tasks
-
-It's important to use exactly these task names as most of them are also being
-used to define dependencies:
-
- `validation`
-    - `Validate Environment`
- `buildprod`
-    - `Build Prod Site`
- `build`
-    - `Build Site`
-    - `Download DB`
- `prepare`
-    - `Import DB`
-    - `Update DB`
-    - `Build Theme`
- `test`
-    - `Test Code Style`
-    - `Test PHPUnit`
-    - `Test Behat`
-    - `Test Backstop`
- `deploy`
-    - `Deploy`
-
 ## Usage

 This chapter is incomplete so far and needs more attention.