#16 GitLab CI/CD for Drupal projects

docs/gitlab/drupal/index.md

@@ -7,7 +7,307 @@ tags:
 - ci/cd
 - drupal
 ---
-## Usage for test and deploy
+# GitLab CI for Drupal pipelines
 
-* You can set the CI/CD variable `COMPOSER_DOWNGRADE` to `1` for using the composer version 1. Default is composer version 2.
-* For authenticating with the Gitlab package repository, you have to set the variable `GITLAB_ACCESS_TOKEN`, which you have to create for the user who runs the pipeline.
+This project contains a number of pre-configured tasks for GitLab CI/CD that
+allow to run very powerful pipelines which contain the following stages:
+
+- validation
+- build
+- prepare
+- test
+- deploy
+
+These pre-configured tasks can easily be included into your Drupal project.
+A GitLab runner on any host can then be configured to run those pipelines. The 
+following chapters describe the necessary steps and also all the options
+available for configuration.
+
+## Configuration
+
+### GitLab Runner
+
+To setup a GitLab runner, they need to be
+[installed first](https://docs.gitlab.com/runner/install) - see also our
+[Ansible role](/ansible/roles/gitlab-runner). Then go to the Drupal project
+on GitLab into **Settings / CI/CD / Runners** and follow the instructions
+there. Make sure that the runner gets tagged with `default` in GitLab.
+
+As a result, a file `/etc/gitlab-runner/config.toml` get created and this
+should be edited to look like this:
+
+```
+[[runners]]
+  name = "Name of runner"
+  url = "https://gitlab.lakedrops.com/"
+  token = "TOKEN"
+  executor = "docker"
+  [runners.cache]
+    [runners.cache.s3]
+    [runners.cache.gcs]
+  [runners.docker]
+    tls_verify = false
+    image = "registry.lakedrops.com/docker/gitlab-drupal-ci:php-7.4"
+    privileged = true
+    disable_entrypoint_overwrite = false
+    oom_kill_disable = false
+    disable_cache = false
+    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
+    cache_dir = "/cache"
+    shm_size = 0
+```
+
+The image is tagged for the PHP version the project uses. For projects with
+different PHP versions a different GitLab runner should be configured or the
+image being overwritten for each task in the `.gitlab-ci.yml` of the respective
+project.
+
+### GitLab Variables
+
+Some variables are required for proper access control and you have to provide
+them in the project configuration by going to **Settings / CI/CD / Variables**
+and adding these variables:
+
+- `SSH_PRIVATE_KEY`: A private key being generated elsewhere just for this 
+  purpose. This is necessary for cloning private Git repositories.
+- `GITLAB_ACCESS_TOKEN`: For authenticating with the LakeDrops Gitlab package
+  repository, you have to create an access tokeb for the user who runs the
+  pipeline and provide it in this variable.
+
+### GitLab CI instructions in Drupal project
+
+The Drupal project should be setup with the
+[Drupal Development Environment](/composer/plugin/drupal-dev-environment)
+composer plugin and then get a `.gitlab-ci.yml` file for the pipeline
+configuration:
+
+```yaml
+variables:
+  COMPOSE_PROJECT_NAME: myproject_$CI_COMMIT_REF_SLUG
+  ENVIRONMENT_NAME: myproject/$CI_COMMIT_REF_NAME
+  THEME_CSS_PATH: web/themes/custom/mytheme/css
+
+include:
+  - project: 'gitlab-ci-cd/drupal'
+    ref: master
+    file: '/test-and-deploy.yml'
+
+Validate Environment:
+  extends: '.prerequisites'
+
+Build Site:
+  extends: '.build'
+
+Download DB:
+  stage: build
+  tags:
+    - default
+  variables:
+    GIT_STRATEGY: none
+  script: |
+    if [[ "$DBREQUIRED" == "yes" ]]; then
+      echo "Scripts have not been developed yet"
+    fi
+  cache: {}
+  dependencies:
+    - 'Validate Environment'
+  artifacts:
+    name: dbdump
+    when: always
+    paths:
+      - ${CI_PROJECT_NAME}.sql
+  except:
+    refs:
+      - tags
+    variables:
+      - $CI_COMMIT_MESSAGE =~ /^Merge tag /i
+      - $CAE
+      - $DISABLE_CI_TESTS
+
+Import DB:
+  extends: '.importdb'
+
+Update DB:
+  extends: '.updatedb'
+
+Build Theme:
+  extends: '.theme'
+  before_script:
+    - cd web/themes/custom/mytheme
+
+Test Code Style:
+  extends: '.codestyle'
+
+Test PHPUnit:
+  extends: '.phpunit'
+
+Test Behat:
+  extends: '.behat'
+
+Test Backstop:
+  extends: '.backstop'
+
+Deploy:
+  stage: deploy
+  tags:
+    - default
+  variables:
+    GIT_STRATEGY: none
+  environment:
+    name: ${ENVIRONMENT_NAME}
+  script:
+    - echo "Scripts have not been developed yet"
+  cache: {}
+  dependencies:
+    - 'Build Theme'
+  except:
+    refs:
+      - tags
+    variables:
+      - $CI_COMMIT_MESSAGE =~ /^Merge tag /i
+      - $CAE
+```
+
+The tasks for downloading the database dump and to deploy the site finally,
+depend on your hosting environment and have not been generalized, unless you're
+using our Ansible environment too, then please refer to
+[Using Ansible to Dump DB](#Using Ansible to Dump DB) and
+[Using Ansible for Deployment](#Using Ansible for Deployment) below. Otherwise,
+you have to write the scripts for those two tasks yourselves.
+
+#### Variables
+
+- `COMPOSE_PROJECT_NAME`
+- `ENVIRONMENT_NAME`
+- `THEME_CSS_PATH`
+- `COMPOSER_DOWNGRADE`: by default, composer 2 is being used. To use composer 1
+  instead, set this variable to 1.
+- `DISABLE_CI_TESTS`: if this variable is set to any value, the tasks in the
+  test stage will be skipped. This is useful e.g. in a development environment
+  where you push and run pipelines often but don't want to run the tests every
+  single time.
+- `CAE`: This variable is being used by the Drupal module 
+  [Config auto export](https://www.drupal.org/project/config_auto_export)
+
+#### Stages and Tasks
+
+It's important to use exactly these task names as most of them are also being
+used to define dependencies:
+
+- `validation`
+    - `Validate Environment`
+- `build`
+    - `Build Site`
+    - `Download DB`
+- `prepare`
+    - `Import DB`
+    - `Update DB`
+    - `Build Theme`
+- `test`
+    - `Test Code Style`
+    - `Test PHPUnit`
+    - `Test Behat`
+    - `Test Backstop`
+- `deploy`
+    - `Deploy`
+
+### Using Ansible
+
+If you also use an [Ansible inventory](/ansible) for your hosts, you can of
+course integrate them into the GitLab pipelines too. To make this work, you
+need a separate GitLab runner tagged with `ansible` and a configuration
+like this:
+
+```
+[[runners]]
+  name = "Name of ansible runner"
+  url = "https://gitlab.lakedrops.com/"
+  token = "TOKEN"
+  executor = "docker"
+  environment = ["DOCKER_AUTH_CONFIG={\"auths\":{\"registry.lakedrops.com\":{\"auth\":\"YOURAUTHTOKEN\"}}}"]
+  [runners.custom_build_dir]
+  [runners.cache]
+    [runners.cache.s3]
+    [runners.cache.gcs]
+  [runners.docker]
+    tls_verify = false
+    hostname = "Ansible-INVENTORYNAME"
+    image = "registry.lakedrops.com/ansible-inventories/INVENTORYNAME:latest"
+    privileged = true
+    disable_entrypoint_overwrite = false
+    oom_kill_disable = false
+    disable_cache = false
+    volumes = ["/home/gitlab-runner/.ssh/id_rsa:/root/.ssh/id_rsa", "/home/gitlab-runner/.a/variables.yml:/root/.ansible/secrets", "/home/gitlab-runner/.a/inventories/INVENTORYNAME:/root
+/.ansible", "/etc/ansible.yml:/etc/ansible.yml", "/var/log/ansible:/var/log/ansible"]
+    pull_policy = "always"
+    shm_size = 0
+```
+
+The used placeholders in this example are:
+
+- `TOKEN`: the token of the GitLab runner being generated during setup.
+- `YOURAUTHTOKEN`: your auth token to the LakeDrops GitLab Docker registry.
+- `INVENTORYNAME`: the name of the Ansible inventory to be used.
+
+To learn more about the Ansible imeages for an inventory and how to setup some
+of the configuration files, please have a look into
+[Docker for Ansible](/docker/ansible).
+
+Once you have the GitLab runner for Ansible setup for the Drupal project,
+you can adjust your GitLab CI configuration:
+
+#### Using Ansible to Dump DB
+
+You can replace the task above with this simple instruction:
+
+```yaml
+Download DB:
+  extends: '.dumpdb'
+```
+
+#### Using Ansible for Deployment
+
+You need a couple of additional variables and then you can replace the deploy
+task above with this simple instruction:
+
+```yaml
+variables:
+  HOST_NAME: myhostname
+  SITE_NAME: mysiteid
+
+Deploy:
+  extends: '.deploy'
+```
+
+The host name needs to be set to how the destination host is known in the
+Ansible inventory, and the site name needs to be set to the ID of the Drupal
+site in your inventory.
+
+## Usage
+
+This chapter is incomplete so far and needs more attention.
+
+### Handling of the database
+
+To build and test your Drupal site prior to deployment, a database with content
+for this project is required and this is handled by the pipeline pretty smart.
+
+First, it needs to be decided, if a fresh dump of the database needs to be
+collected or if the already existing database from the previous pipeline run
+can be re-used. Here is how the prepared pipelines make that decision:
+
+A fresh database is being pulled if one of the following conditions apply,
+tested in the given order:
+
+- if the pipeline run for the master branch
+- if the commit message contains the string `[PULL_DB]`
+- if the variable `PULL_DB` is set to `yes`
+- if no database container from a previous pipeline exists
+- if the database from the previous pipeline doesn't contain any user data yet
+
+If a new database is required, the task `Download DB` in the `build` stage will
+create a dump and make it available as an artifact for subsequent tasks. The 
+task `Import DB` will then import that dump in the `prepare` stage.
+
+Otherwise, those two tasks will be skipped and the task `Update DB` will be
+executed in the `prepare` stage instead.