Add devops alm documents

.gitignore

 /site/
 /.ahoy.l3d
 /.env
-/docs/diagrams/out/
+/docs/diagrams/uml/out/

docs/devops/alm/alm.md

+# Application Lifecycle Management (ALM)
+
+The ALM is one of our most important topics in our development strategy. Here, you gain an insight of our company's
+philosophy concerning lean and professional software development.
+
+This document describes the different roles of our configuration management. Principles, features and tools are
+explained in detail. This document is linked to our [Hosting & Maintenance](./hosting_maintain) document, where we also follow our
+philosophy.
+
+## Roles
+
+To satisfy the principle of separation of concerns, we have defined specific roles, which are involved
+in our ALM. Every role has different requirements and expectations from the environment they are working in.
+
+The roles are:
+
+### Customer
+
+First of all, the ALM is designed to provide the biggest benefit for our customers. The customer expects working
+software and wants to see results as fast as possible. This leads to short feedback cycles, a faster progress and
+a good communication between the customer and ourselves.
+
+So, the customer and their product can have a short time to market.
+
+### Developer
+
+This role has the most interactions with the ALM environment. A developer has the following requirements:
+
+* fast setup of a developer environment
+* enable collaboration in a developer team
+* a proper way to communicate with the customer
+* of course, develop the software
+* fast execution of tests
+* short feedback cycles with the building system
+
+### Builder
+
+Although in our case the builder role is a machine, it is described here as well. The reason is we want the builder to work
+completely automated. Here, the focus is on consistency. So, e.g., the tests
+should run as fast as possible, but since this process is automated, it is not that time critical. The builder logs all the build steps and satisfy the fail fast principle. This means the build immediately stops, 
+when something is wrong. Those build failures must be reported to the developers, so they can fix them as 
+soon as possible.
+
+## Overview
+
+Here you can see the complete process of our ALM and how it satisfies all the goals and requirements of each
+role described above.
+
+![ALM](/diagrams/alm/alm.svg)
+
+First, we start with the role developer in the bottom right. One of the most important interactions for a developer
+is with the repository. We use GitLab for our entire configuration management. Everything a developer needs to work with, is stored here:
+
+* Source Code in GIT repositories
+* Developer Tools in Docker Images
+* Documentation in Wiki projects
+
+We also use GitLab as our ticket system, where we can interact with the customer and other teammates. For our Drupal
+development the toolset allows us to pull data from the live system. This allows the developer to work with the real
+data or to reproduce bugs as fast as possible. The developer has the total control of the process. They can work and run the tests on their local machine
+and when they decide to push a change, the builder role gets triggered. The builder takes the same Docker images from GitLab (without the developer tools)
+and starts to build the project. This is important, because we follow the DRY (don't repeat yourself) principle, and the builder should
+use the same tools as the developer. The building process covers the following steps:
+
+* Build
+* Prepare
+* Test (Unit, Integration, Visual Behaviour, Code Style, etc.)
+
+If all the phases are passed, the site gets deployed depending on the branch, like "Feature A", "Test" or "Live". This is completely automated. The
+deployment scripts are written with Ansible, which are also stored in GitLab projects. So, after a build was successful, the customer can see the results
+immediately on the specific version. For instance, if we work on a feature, and a product increment is ready, the customer can use and provide feedback in
+a little while.
+
+To summarize our process the following chapters provide a comprehensive list of our features, principles and the tools in use.
+
+## Features and Principles
+
+### Infrastructure as Code
+
+This is one of our key concepts. All our code for creating our infrastructure and tools is stored in Ansible scripts
+in our GitLab. These scripts can always be executed through jobs in a build pipeline. 
+
+### Provide customer benefit as fast as possible
+
+After a product increment or an entire feature is implemented, the customer can see the result in a few minutes. The customer has access to
+our GitLab projects which include the ticket system and pipelines.
+
+### Provide Quality Software
+
+Of course developers strive to provide good software, and they can write and execute the tests on their local machine.
+But we go a step further. In our build pipeline every test gets executed automatically and fails if something goes wrong. 
+If this happens, nothing gets deployed, so we can always provide a running system. 
+
+### Single point of entry
+
+The customer just can go the specific GitLab project and contact us by opening a ticket. They can also follow the progress in Kanban Boards or even
+in the build pipelines.
+
+The same goes for the developers. They are working on the code base, which is managed by GitLab. Even the tools for development are managed by GitLab.
+
+### Keep Drupal Configuration up to date
+
+As you can see in the overview picture above, there is a connection between the live configuration and the GitLab repository. If someone updates
+the configuration on the website, the changes are pushed to the Git repository. This assures an actual system and retracing the changes that are made.
+
+### Open Source Toolset
+
+You do not need a paid licence for using Drupal of course. We want to keep this philosophy for our tools as well. All deployment scripts and developer tools are 
+free to use for everyone. This allows to raise the quality, and the number of features, so everybody can benefit from.
+ 
+### Easy entry area for Drupal developers
+
+Developers just need Docker on their local machine. The development tool L3D, which is open source, creates all the Docker containers needed for
+developing Drupal. This also includes debugging and testing. Now, the developer can use the IDE of their choice.
+
+## Tools
+
+Here you see our tools, which are involved in our ALM. As you can see, we make heavy use of GitLab. This allows us to maintain just one tool,
+which provides a huge set of features and a single point of entry for our customers. We also want to satisfy our principle of separation of concerns. Every tool
+is designed to achieve a specific goal and just using it for this.
+
+| Function | Tool |
+| --- | --- |
+| Repo / Versioning | [GitLab](https://about.gitlab.com) |
+| Issues / Communication | [GitLab](https://about.gitlab.com) |
+| Agile Planning | [GitLab](https://about.gitlab.com) |
+| CI/CD Pipelines | [GitLab](https://about.gitlab.com) |
+| Drupal Development | [Docker](https://www.docker.com) |
+| Deployment | [Ansible](https://www.ansible.com) |

docs/devops/alm/hosting_maintain.md

+# Hosting and Maintenance
+
+This document gives a comprehensive overview of all important concepts, components and tools in use. As you will see
+one of our key expertise is to automate monitoring to provide an excellent quality concerning stability.
+
+This document is also linked to the document [ALM](alm), where you find similar philosophies and principles.
+
+## Overview
+
+![Overview](/diagrams/alm/overview.svg)
+
+The following features explain our environment in a more detailed fashion.
+
+### Infrastructure as Code
+
+This is one of our key concepts. All our code for creating our infrastructure and tools is stored in Ansible scripts
+in our GitLab. These scripts can always be executed through jobs in a build pipeline. It is also a key part of
+our recovery concept described later in this document.
+
+### Live Monitoring
+
+By using sensors and heartbeats, we can monitor a host by checking:
+
+* If the host is running
+* CPU
+* Memory Usage
+* Disk I/O
+* Network
+
+We use the tool Netdata, several thousand metrics are monitored every second, e.g.:
+
+* MySQL
+* Docker
+* Apache
+* etc.
+
+### Extensive Logging
+
+As you can see in the picture above, we collect the logs from every host and from every application in use. This is a huge amount of data,
+so we use Elasticsearch for analysis. This analysis can also result in an alert, which is sent to our alerting system described below. 
+
+### High-Quality Analysis
+
+With Netdata and logging in place we can provide a comprehensive analysis of all our components. It allows us to retrace problems or even
+avoid problems, which might appear in the future.
+
+### Alert Deduplication
+
+As seen in the picture this component has four incoming channels:
+
+* Result of the Log Analysis
+* Host sensors 
+* Host heartbeats
+* Applications
+
+This component is critical because it acts as a single pool of alerts for the DevOps staff.
+We want to avoid duplications to get a clear overview of what has happened.
+
+### Automatic Ticket Creation
+
+Our alert deduplication component will create tickets in GitLab automatically, when something goes wrong. This allows us to react as fast as
+possible to take care of the potential problem.
+
+### CI / CD Pipelines and Communication
+
+The tasks defined in the pipelines can be triggered from various sources, like:
+* Schedule, e.g. running every night
+* Action of a developer in Git (our VCS)
+* Command from the communication platform. We use the ChatOps feature in Mattermost.
+
+The tasks are developed once to satisfy our goal of DRY (don't repeat yourself). These tasks can be used from different triggers.
+
+### Result of CI / CD Pipelines
+
+The pipelines have three main goals:
+
+* Provisioning: Covers all the tools of the host
+* Maintenance: Covers the configuration of the applications
+* Deployment: Install and update the application
+
+### Manage Configuration in Repository
+
+Configuration of a host is managed by the Git repositories. This allows us to easily keep track and recover the system. This concept is
+explained in the next chapter.
+
+## Recovery
+
+The following components are backed up by the 3-2-1 philosophy and can be recovered. 
+
+![Recovery](/diagrams/alm/recovery.svg)
+
+Note: The DB and User Data recovery duration cannot be determined due to the amount of data and network speed.
+
+## Tools
+
+| Function | Tool |
+| --- | --- |
+| Repo / Versioning | [GitLab](https://about.gitlab.com) |
+| Issues | [GitLab](https://about.gitlab.com) |
+| CI/CD Pipelines | [GitLab](https://about.gitlab.com) |
+| Communication | [Mattermost](https://mattermost.com/) |
+| Provisioning / Deployment / Maintenance | [Ansible](https://www.ansible.com) |
+| Alert Deduplication | [Alerta](https://alerta.io) |
+| Sensors | [Netdata](https://www.netdata.cloud/) and [Beats](https://www.elastic.co/beats/) |
+| Heartbeats | [curl](https://curl.se/) |
+| Log Collection | [TD-Agent](https://docs.fluentd.org/) |
+| Log Aggregation | [Elasticsearch](https://www.elastic.co/elasticsearch/) |
+| Log Visualizing | [Kibana](https://www.elastic.co/kibana) |
+| Log Analysis | [ElastAlert](https://elastalert.readthedocs.io/en/latest/) |
+| Ping | [Netdata API](https://learn.netdata.cloud/docs) |
+| Backup | [Borg Backup](https://www.borgbackup.org/) |

docs/devops/alm/tools.md

+# Tools used for DevOps
+
+| Function | Tool |
+| --- | --- |
+| Repo / Versioning | GitLab |
+| Issues | GitLab |
+| CI/CD Pipelines | GitLab |
+| Communication | Mattermost |
+| Provisioning / Deployment / Maintenance | Ansible |
+| Alert Deduplication | Alerta |
+| Sensors | Netdata + Beats |
+| Heartbeats | curl |
+| Log Collection | TD-Agent |
+| Log Aggregation | ElasticSearch |
+| Log Visualizing | Kibana |
+| Log Analyzing | ElastAlert |
+| Ping | Netdata API |

docs/diagrams/src/test1.puml

-@startuml
-
-  Goofy ->  MickeyMouse: calls
-  Goofy <-- MickeyMouse: responds
-
-@enduml

docs/index.md

-# Welcome to DevOps Tools
-
-For full documentation visit [mkdocs.org](https://mkdocs.org).
-
-![file](diagrams/out/test1.svg)
+# DevOps Tools Documentation

mkdocs.yml

@@ -10,121 +10,126 @@ plugins:
       render: "server"
       server: "http://www.plantuml.com/plantuml"
       output_format: "svg"
-      diagram_root: "docs/diagrams"
+      diagram_root: "docs/diagrams/uml"
       output_folder: "out"
       input_folder: "src"
       input_extensions: ""
 nav:
   - Home: index.md
   - Docker:
-      - Home: docker/index.md
-      - GitLab Drupal CI: docker/gitlab-drupal-ci/index.md
-      - L3D: docker/l3d/index.md
-      - Maven: docker/maven-build/index.md
-      - Node: docker/node/index.md
+    - Home: docker/index.md
+    - GitLab Drupal CI: docker/gitlab-drupal-ci/index.md
+    - L3D: docker/l3d/index.md
+    - Maven: docker/maven-build/index.md
+    - Node: docker/node/index.md
   - Composer:
-      - Home: composer/index.md
-      - Libraries:
-          - Composer.json Utilities: composer/library/composer-json-utils/index.md
-          - Docker Traefik: composer/library/docker-traefik/index.md
-      - Plugins:
-          - Ahoy: composer/plugin/ahoy/index.md
-          - Behat for Drupal: composer/plugin/b4d/index.md
-          - Docker for Drupal: composer/plugin/d4d/index.md
-          - Drupal 8 SASS Theme: composer/plugin/d8-sass-theme/index.md
-          - Drupal 8 Scaffold: composer/plugin/d8-scaffold/index.md
-          - Drupal 9 Scaffold: composer/plugin/d9-scaffold/index.md
-          - Dorgflow: composer/plugin/dorgflow/index.md
-          - Drupal Development Environment: composer/plugin/drupal-dev-environment/index.md
-          - Drupal Environment: composer/plugin/drupal-environment/index.md
-      - Project:
-          - Drupal 8: composer/project/d8/index.md
-          - Drupal 8 Commerce: composer/project/d8-commerce/index.md
-          - Drupal 8 Commerce Demo: composer/project/d8-commerce-demo/index.md
-          - Drupal 9: composer/project/d9/index.md
+    - Home: composer/index.md
+    - Libraries:
+      - Composer.json Utilities: composer/library/composer-json-utils/index.md
+      - Docker Traefik: composer/library/docker-traefik/index.md
+    - Plugins:
+      - Ahoy: composer/plugin/ahoy/index.md
+      - Behat for Drupal: composer/plugin/b4d/index.md
+      - Docker for Drupal: composer/plugin/d4d/index.md
+      - Drupal 8 SASS Theme: composer/plugin/d8-sass-theme/index.md
+      - Drupal 8 Scaffold: composer/plugin/d8-scaffold/index.md
+      - Drupal 9 Scaffold: composer/plugin/d9-scaffold/index.md
+      - Dorgflow: composer/plugin/dorgflow/index.md
+      - Drupal Development Environment: composer/plugin/drupal-dev-environment/index.md
+      - Drupal Environment: composer/plugin/drupal-environment/index.md
+    - Project:
+      - Drupal 8: composer/project/d8/index.md
+      - Drupal 8 Commerce: composer/project/d8-commerce/index.md
+      - Drupal 8 Commerce Demo: composer/project/d8-commerce-demo/index.md
+      - Drupal 9: composer/project/d9/index.md
   - Ansible:
-      - Home: ansible/index.md
-      - Wiki:
-          - Home: ansible/wiki/index.md
-          - Attacks: ansible/wiki/attacks.md
-          - Crontabs: ansible/wiki/configure-crontabs.md
-          - Desaster recovery: ansible/wiki/desaster-recovery.md
-          - Drupal Apache update: ansible/wiki/drupal-apache-update.md
-          - Drupal deployment: ansible/wiki/drupal-deployment.md
-          - Drush fetch aliases: ansible/wiki/drush-fetch-aliases.md
-          - Firewall: ansible/wiki/firewall.md
-          - GitLab CI configuration: ansible/wiki/gitlab-ci-configuration.md
-          - GitLab CI introduction: ansible/wiki/gitlab-ci-introduction.md
-          - GitLab CI prerequisites: ansible/wiki/gitlab-ci-prerequisites.md
-          - HaProxy custom blacklists: ansible/wiki/haproxy-custom-blacklists.md
-          - HaProxy quick update: ansible/wiki/haproxy-quick-update.md
-          - LetsEncrypt: ansible/wiki/letsencrypt.md
-          - Resources: ansible/wiki/resources.md
-          - Swapfile: ansible/wiki/swapfile.md
-          - Add new user: ansible/wiki/user-add-new.md
-          - Set user password: ansible/wiki/user-set-password.md
-          - Varnish quick update: ansible/wiki/varnish-quick-update.md
-          - Backup:
-              - Home: ansible/wiki/backup/backup.md
-              - Intro: ansible/wiki/backup/introduction.md
-              - Backup to Google: ansible/wiki/backup/backup-google.md
-              - Backup to SpiderOak: ansible/wiki/backup/backup-spideroak.md
-              - Restore: ansible/wiki/backup/restore.md
-              - Restore from Google: ansible/wiki/backup/restore-google.md
-              - Restore from SpiderOak: ansible/wiki/backup/restore-spideroak.md
-          - ELK:
-              - Home: ansible/wiki/elk/introduction.md
-              - Fluentd: ansible/wiki/elk/fluentd.md
-              - Kibana: ansible/wiki/elk/kibana.md
-          - Hosts:
-              - Add new host: ansible/wiki/hosts/add.md
-              - JiffyBox: ansible/wiki/hosts/jiffybox.md
-              - Prevent reboot: ansible/wiki/hosts/prevent-reboot.md
-          - Inventory:
-              - Add new inventory: ansible/wiki/inventory/add.md
-              - ChatOps: ansible/wiki/inventory/chatops.md
-              - Pipelines: ansible/wiki/inventory/pipelines.md
-          - Monitoring:
-              - Home: ansible/wiki/monitoring/monitoring.md
-              - Introduction: ansible/wiki/monitoring/introduction.md
-              - Alerts: ansible/wiki/monitoring/alerts.md
-              - Alerts to ELK: ansible/wiki/monitoring/alerts-elk.md
-              - Alerts on NetData: ansible/wiki/monitoring/alerts-netdata.md
-              - Alerts on uptime: ansible/wiki/monitoring/alerts-uptime.md
-          - OpenSSL:
-              - CSR: ansible/wiki/openssl/csr.md
-          - Tips:
-              - Signed Git commits: ansible/wiki/tips/signed-git-commits.md
-      - Plugins:
-          - Fluentd: ansible/plugins/fluentd/index.md
-          - ServerDensity: ansible/plugins/serverdensity/index.md
-      - Roles:
-          - Apache: ansible/roles/apache/index.md
-          - Borg Backup: ansible/roles/borgbackup/index.md
-          - Composer: ansible/roles/composer/index.md
-          - Discourse: ansible/roles/discourse/index.md
-          - ElastAlert: ansible/roles/elastalert/index.md
-          - Elasticsearch: ansible/roles/elasticsearch/index.md
-          - Fail2Ban: ansible/roles/fail2ban/index.md
-          - Fluentd: ansible/roles/fluentd/index.md
-          - GitLab: ansible/roles/gitlab/index.md
-          - HaProxy: ansible/roles/haproxy/index.md
-          - Heartbeat: ansible/roles/heartbeat/index.md
-          - JailKit: ansible/roles/jailkit/index.md
-          - Keycloak: ansible/roles/keycloak/index.md
-          - LetsEncrypt: ansible/roles/letsencrypt/index.md
-          - MySQL: ansible/roles/mysql/index.md
-          - Nextcloud: ansible/roles/nextcloud/index.md
-          - Oracle: ansible/roles/oracle/index.md
-          - Packetbeat: ansible/roles/packetbeat/index.md
-          - PHP: ansible/roles/php/index.md
-          - ServerDensity: ansible/roles/serverdensity/index.md
-          - SpiderOak ONE: ansible/roles/spideroak/index.md
-          - User Management: ansible/roles/user-management/index.md
-          - VPN: ansible/roles/vpn/index.md
-          - Zammad: ansible/roles/zammad/index.md
+    - Home: ansible/index.md
+    - Wiki:
+      - Home: ansible/wiki/index.md
+      - Attacks: ansible/wiki/attacks.md
+      - Crontabs: ansible/wiki/configure-crontabs.md
+      - Desaster recovery: ansible/wiki/desaster-recovery.md
+      - Drupal Apache update: ansible/wiki/drupal-apache-update.md
+      - Drupal deployment: ansible/wiki/drupal-deployment.md
+      - Drush fetch aliases: ansible/wiki/drush-fetch-aliases.md
+      - Firewall: ansible/wiki/firewall.md
+      - GitLab CI configuration: ansible/wiki/gitlab-ci-configuration.md
+      - GitLab CI introduction: ansible/wiki/gitlab-ci-introduction.md
+      - GitLab CI prerequisites: ansible/wiki/gitlab-ci-prerequisites.md
+      - HaProxy custom blacklists: ansible/wiki/haproxy-custom-blacklists.md
+      - HaProxy quick update: ansible/wiki/haproxy-quick-update.md
+      - LetsEncrypt: ansible/wiki/letsencrypt.md
+      - Resources: ansible/wiki/resources.md
+      - Swapfile: ansible/wiki/swapfile.md
+      - Add new user: ansible/wiki/user-add-new.md
+      - Set user password: ansible/wiki/user-set-password.md
+      - Varnish quick update: ansible/wiki/varnish-quick-update.md
+      - Backup:
+        - Home: ansible/wiki/backup/backup.md
+        - Intro: ansible/wiki/backup/introduction.md
+        - Backup to Google: ansible/wiki/backup/backup-google.md
+        - Backup to SpiderOak: ansible/wiki/backup/backup-spideroak.md
+        - Restore: ansible/wiki/backup/restore.md
+        - Restore from Google: ansible/wiki/backup/restore-google.md
+        - Restore from SpiderOak: ansible/wiki/backup/restore-spideroak.md
+      - ELK:
+        - Home: ansible/wiki/elk/introduction.md
+        - Fluentd: ansible/wiki/elk/fluentd.md
+        - Kibana: ansible/wiki/elk/kibana.md
+      - Hosts:
+        - Add new host: ansible/wiki/hosts/add.md
+        - JiffyBox: ansible/wiki/hosts/jiffybox.md
+        - Prevent reboot: ansible/wiki/hosts/prevent-reboot.md
+      - Inventory:
+        - Add new inventory: ansible/wiki/inventory/add.md
+        - ChatOps: ansible/wiki/inventory/chatops.md
+        - Pipelines: ansible/wiki/inventory/pipelines.md
+      - Monitoring:
+        - Home: ansible/wiki/monitoring/monitoring.md
+        - Introduction: ansible/wiki/monitoring/introduction.md
+        - Alerts: ansible/wiki/monitoring/alerts.md
+        - Alerts to ELK: ansible/wiki/monitoring/alerts-elk.md
+        - Alerts on NetData: ansible/wiki/monitoring/alerts-netdata.md
+        - Alerts on uptime: ansible/wiki/monitoring/alerts-uptime.md
+      - OpenSSL:
+        - CSR: ansible/wiki/openssl/csr.md
+      - Tips:
+        - Signed Git commits: ansible/wiki/tips/signed-git-commits.md
+    - Plugins:
+      - Fluentd: ansible/plugins/fluentd/index.md
+      - ServerDensity: ansible/plugins/serverdensity/index.md
+    - Roles:
+      - Apache: ansible/roles/apache/index.md
+      - Borg Backup: ansible/roles/borgbackup/index.md
+      - Composer: ansible/roles/composer/index.md
+      - Discourse: ansible/roles/discourse/index.md
+      - ElastAlert: ansible/roles/elastalert/index.md
+      - Elasticsearch: ansible/roles/elasticsearch/index.md
+      - Fail2Ban: ansible/roles/fail2ban/index.md
+      - Fluentd: ansible/roles/fluentd/index.md
+      - GitLab: ansible/roles/gitlab/index.md
+      - HaProxy: ansible/roles/haproxy/index.md
+      - Heartbeat: ansible/roles/heartbeat/index.md
+      - JailKit: ansible/roles/jailkit/index.md
+      - Keycloak: ansible/roles/keycloak/index.md
+      - LetsEncrypt: ansible/roles/letsencrypt/index.md
+      - MySQL: ansible/roles/mysql/index.md
+      - Nextcloud: ansible/roles/nextcloud/index.md
+      - Oracle: ansible/roles/oracle/index.md
+      - Packetbeat: ansible/roles/packetbeat/index.md
+      - PHP: ansible/roles/php/index.md
+      - ServerDensity: ansible/roles/serverdensity/index.md
+      - SpiderOak ONE: ansible/roles/spideroak/index.md
+      - User Management: ansible/roles/user-management/index.md
+      - VPN: ansible/roles/vpn/index.md
+      - Zammad: ansible/roles/zammad/index.md
   - GitLab:
-      - Home: gitlab/index.md
-      - Ansible: gitlab/ansible/index.md
-      - Composer: gitlab/composer/index.md
-      - Drupal: gitlab/drupal/index.md
+    - Home: gitlab/index.md
+    - Ansible: gitlab/ansible/index.md
+    - Composer: gitlab/composer/index.md
+    - Drupal: gitlab/drupal/index.md
+  - DevOps:
+    - ALM:
+      - Introduction: devops/alm/alm.md
+      - Hosting and Monitoring: devops/alm/hosting_maintain.md
+      - Tools: devops/alm/tools.md