diff --git a/.gitignore b/.gitignore index e653421831c07c54884e0774e7f1126f204261d1..f98c5f34320cd1c9db0598979be245eac54feb7d 100644 --- a/.gitignore +++ b/.gitignore @@ -14,3 +14,4 @@ /.idea/vcs.xml .gitignore /site/ +/.idea/git_toolbox_blame.xml diff --git a/docs/gitlab/drupal/index.md b/docs/gitlab/drupal/index.md index 94969c60417f14b71586b815f46bc9b5687a6cbb..33c8a900bdec60e5d72c4b19f01a3b53a729eb2a 100644 --- a/docs/gitlab/drupal/index.md +++ b/docs/gitlab/drupal/index.md @@ -12,18 +12,373 @@ tags: This project contains a number of pre-configured tasks for GitLab CI/CD that allow to run very powerful pipelines which contain the following stages: -- validation +- .pre (predefined) +- validate +- execute +- build (predefined) - buildprod -- build - prepare -- test -- deploy +- test (predefined) +- deploy (predefined) +- release +- finalize +- postprocess +- .post (predefined) + +We have a predefined `execute` job called `LakeDrops`, which looks like this: + +- execute + - LakeDrops + - buildprod + - Build Prod Site + - Build Prod Site NG + - build + - Build Site + - Build Theme + - Check 4 Outdated Packages + - Check 4 Security Vulnerability Advisories + - Download DB + - Download DB NG + - prepare + - Import DB + - Import DB NG + - Import Local DB + - Update DB + - test + - Test Code Style + - Test PHPUnit + - Test Backstop + - Test Cypress E2E + - finalize + - Shut Down Docker Project These pre-configured tasks can easily be included into your Drupal project. A GitLab runner on any host can then be configured to run those pipelines. The following chapters describe the necessary steps and also all the options available for configuration. +## Stages +The following chapters describes all stages and their jobs. The files: + +- [lakedrops.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/lakedrops.yml?ref_type=heads). +- [test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads). + +### .pre +To prepare and check, if the environment is good to go, we have a `.pre` stage, which contains the +following jobs: + +#### Sanity Checks + +The sanity check gets important when you are using [Crowdsec](https://www.crowdsec.net/). +It contains two little scripts, that check: + +- if the `crowdsec.settings.yml` is in the config split and +- if it is not enabled in the `core.extensions.yml`. + +Otherwise, it is conflicting with you local settings, since `CrowdSec` +is a tool for production sites. + +Except: + +- Commit message contains "Merge tag" +- Commit message contains "Merge branch" +- When the job is triggered by config auto export (CAE) + +#### Validate Environment + +This job extends the `.prerequisites` in +[test_deploy.yml](https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/test-and-deploy.yml?ref_type=heads). + +This job executes a script, where several [variables](#variables) are checked and set accordingly. + +Except: + +- Commit message contains "Merge tag" +- Commit message contains "Merge branch" +- When the job is triggered by config auto export (CAE) + +### Validate + +This stage is currently not officially used. + +### Execute + +This stage executes the build itself. + +### Build Prod + +This stage is responsible for building the production website, which we want to finally deploy. + +#### Build Prod Site + +First we have to do some preparation. This is done in the `before_script` section. + +- [.prepareaccess](#prepareaccess) +- [.preparecomposerplugins](#preparecomposerplugins) + +We start a docker image for php with the specifies versions: + +- `PHP_MAJOR_VERSION` +- `PHP_MINOR_VERSION` + +In the script section we first update the environment by executing `/usr/local/bin/update-env`. + +After that we install the production site with `composer install` **without** the tools for development. That is pretty +it. We also add `drupal/core-vendor-hardening`. + +The entire installation is saved as a build artefact called `build-prod` where all `git` files are excluded. + +#### Build Prod Site NG + +First we have to do some preparation. This is done in the `before_script` section. + +- [.prepareaccess](#prepareaccess) +- [.preparecomposerplugins](#preparecomposerplugins) + +We start a docker image for php with the specifies versions: + +- `PHP_MAJOR_VERSION` +- `PHP_MINOR_VERSION` + +In the script part we install two of our LakeDrops modules: + +- [lakedrops/drupal-environment](../../composer/plugin/drupal-environment) +- [lakedrops/docker4drupal](../../composer/plugin/d4d) + +After that we execute: + +- `composer lakedrops:config` +- `composer lakedrops:docker4drupal` + +The entire installation is saved as a build artefact called `build-prod-ng` where all `git` files are excluded. + +Rules, when the job should run: + +- `$DISABLE_DEPLOYMENT_LOCAL` != "1" + +### Build + +Here, we define jobs, which build the Drupal website and does some additional checks. This stage is +responsible for the database downloads as well. + +#### Build Site + +We start a docker image for php with the specifies versions: + +- `PHP_MAJOR_VERSION` +- `PHP_MINOR_VERSION` + +After that, we start to build the Drupal site with [Composer](https://getcomposer.org/): + +Now we execute a `composer install` twice to ensure that all patches get applied correctly. After +that we setup our own `LakeDrops` tools by `composer lakedrops:scaffold` and +`composer lakedrops:docker4drupal`. + +The next step is to pull all required [Docker](https://www.docker.com/) images +with [Docker Compose](https://docs.docker.com/compose/) adn start the services. + +After everything has started, we create some directories, e.g. for styles, logs and JavaScript. + +The entire installation is saved as a build artefact called `build` where all `git` files are excluded. + +Rules, when the job should run: + +- the tests are enabled by `$TESTSDISABLED` == "no" + +#### Build Theme + +This job creates the required files for the styling of the website. + +The following jobs must run before successfully: + +- [Build Prod Site](#build-prod-site) + + + +#### Download DB + +This job executes a script, that removes the SQL file `$]PROJECT_NAME}.sql`, if it exists already. +After that an Ansible script is executed, which dumbs the current Drupal database of the live stage. + +The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`. + +Rules, when the job must not run: + +- `$DISABLE_DEPLOYMENT` != "0" +- `$DISABLE_DEPLOYMENT_LOCAL` != "1" +- `$DBREQUIRED` == "no" +- `$INITIALINSTALL` == "yes" +- `$TESTSDISABLED` == "yes" +- `$LOCALDBFILE` != "none" + +Rules, when the job should run: + +- the tests are enabled by `$DISABLE_CI_TESTS` != "1" + +#### Download DB NG + +This is very similar to [Download DB](#download-db), but does not use Ansible. +It uses Docker Compose instead to execute the `drush sql:dump` command +to dumb the current live database to the file `$]PROJECT_NAME}.sql`. + +The file is saved as a build artefact, which contains `$]PROJECT_NAME}.sql`. + +Rules, when the job must not run: + +- `$DISABLE_DEPLOYMENT` != "0" +- `$DISABLE_DEPLOYMENT_LOCAL` != "0" +- `$DBREQUIRED` == "no" +- `$INITIALINSTALL` == "yes" +- `$TESTSDISABLED` == "yes" +- `$LOCALDBFILE` != "none" + +Rules, when the job should run: + +- the tests are enabled by `$DISABLE_CI_TESTS` != "1" + +### Prepare + +This stage is used to prepare the system for testing. This stage is also used for internal purpose +and for proprietary tools. + +#### Import DB + +This job extends `.importdb`, which itself defines the +anchor [import_db_default](#import-db-default-anchor). + +The following jobs must run before successfully: + +- [Build Site](#build-site) +- [Download DB](#download-db) + +Rules, when the job must not run: + + - `$DISABLE_DEPLOYMENT` != "0" + - `$DISABLE_DEPLOYMENT_LOCAL` != "1" + - `$DBREQUIRED` == "no" + - `$INITIALINSTALL` == "yes" + - `$TESTSDISABLED` == "yes" + - `$LOCALDBFILE` != "none" + +Rules, when the job should run: + +- the tests are enabled by `$DISABLE_CI_TESTS` != "1" + +##### Import DB Default Anchor + +We start a docker image for php with the specifies versions: + +- `PHP_MAJOR_VERSION` +- `PHP_MINOR_VERSION` + +A script copies the available SQL file into the PHP container, drops the current DB and imports the file. +If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. After that the +`UPDATE_DB_COMMAND` gets called as well as a cache reset. + +#### Import DB NG + +This job extends `.importdbNG`, which itself defines the +anchor [import_db_default](#import-db-default-anchor). + +The following jobs must run before successfully: + +- [Build Site](#build-site) +- [Download DB NG](#download-db-ng) + +Rules, when the job must not run: + +- `$DISABLE_DEPLOYMENT` != "0" +- `$DISABLE_DEPLOYMENT_LOCAL` != "0" +- `$DBREQUIRED` == "no" +- `$INITIALINSTALL` == "yes" +- `$TESTSDISABLED` == "yes" +- `$LOCALDBFILE` != "none" + +Rules, when the job should run: + +- the tests are enabled by `$DISABLE_CI_TESTS` != "1" + +#### Import Local DB + +This job extends `.importdbLocal`, which itself defines the +anchor [import_db_default](#import-db-default-anchor). + +The following jobs must run before successfully: + +- [Build Site](#build-site) + +A before script is executed, which copies the `${LOCALDBFILE}` to `${PROJECT_NAME}.sql.gz` and +unzips that file. + +Rules, when the job must not run: + +- `$DBREQUIRED` == "no" +- `$TESTSDISABLED` == "yes" +- `$LOCALDBFILE` != "none" + +Rules, when the job should run: + +- the tests are enabled by `$DISABLE_CI_TESTS` != "1" + +#### Update DB + +We start a docker image for php with the specifies versions: + +- `PHP_MAJOR_VERSION` +- `PHP_MINOR_VERSION` + +If the variable `SKIPCONFIGIMPORT` is `no`, a `drush config-import` gets executed as well. +After that the `UPDATE_DB_COMMAND` gets called as well as a cache reset. + +The following jobs must run before successfully: + +- [Build Site](#build-site) + +Rules, when the job must not run: + +- `$DBREQUIRED` == "yes" +- `$DBUPDREQUIRED` == "no" +- `$INITIALINSTALL` == "yes" +- `$TESTSDISABLED` == "yes" + +Rules, when the job should run: + +- the tests are enabled by `$DISABLE_CI_TESTS` != "1" + +### Deploy + +This stage handles the deployment of the produced build artefact. + +#### Deploy + +The following jobs must run before successfully: + +- [Build Prod Site](#build-prod-site) +- [Build Theme](#build-theme) + +This job executes an [Ansible](https://www.ansible.com/) script, which does the deployment. + +Rules, when the job must not run: + +- `$DISABLE_DEPLOYMENT` != "0" +- `$DISABLE_DEPLOYMENT_LOCAL` != "1" + +#### Deploy NG + +The following jobs must run before successfully: + +- [Build Prod Site](#build-prod-site) +- [Build Prod Site NG](#build-prod-site-ng) +- [Build Theme](#build-theme) + +### Internal Stages + +Here we have some internal stages, which cannot be executed alone. The purpose is to +collect repeating execution. + +#### .prepareaccess +#### .preparecomposerplugins + ## Configuration ### GitLab Runner @@ -187,7 +542,7 @@ you have to write the scripts for those two tasks yourselves. - `ENVIRONMENT_NAME`: the environment name for the GitLab UI. - `THEME_CSS_PATH`: the relative path to the theme's css path where the generated artefact can be found. -- `PHP_MAJOR_VERSION`: the major PHP version, defaults to 7. +- `PHP_MAJOR_VERSION`: the major PHP version, defaults to 10. - `PHP_MINOR_VERSION`: the minor PHP version, default to 4. - `COMPOSER_DOWNGRADE`: by default, composer 2 is being used. To use composer 1 instead, set this variable to 1. @@ -201,6 +556,8 @@ you have to write the scripts for those two tasks yourselves. test stage will be skipped. This is useful e.g. in a development environment where you push and run pipelines often but don't want to run the tests every single time. +- `$DISABLE_GITLAB_CI_TESTS`: if this variable is set to "1", no GitLab Tests will be + executed, see: [GitLab Templates](https://www.drupal.org/project/gitlab_templates). - `DISABLE_CI_TEST_BACKSTOP`: if this variable is set to any value, the visual regression tests with Backstop will be skipped. - `DISABLE_CI_TEST_BEHAT`: if this variable is set to any value, the Behat @@ -217,30 +574,6 @@ you have to write the scripts for those two tasks yourselves. completely and built from scratch. Can also be used as `[RESET_LOCALE]` in the Git commit message. -#### Stages and Tasks - -It's important to use exactly these task names as most of them are also being -used to define dependencies: - -- `validation` - - `Validate Environment` -- `buildprod` - - `Build Prod Site` -- `build` - - `Build Site` - - `Download DB` -- `prepare` - - `Import DB` - - `Update DB` - - `Build Theme` -- `test` - - `Test Code Style` - - `Test PHPUnit` - - `Test Behat` - - `Test Backstop` -- `deploy` - - `Deploy` - ## Usage This chapter is incomplete so far and needs more attention.