Improve pipeline structure for develop and main branches of Drupal sites

Current structure

For a Drupal site, there is a develop and a main branch. They depend upon many released other projects from drupal.org, packagist.org and sometimes custom projects on other repositories.

Daily (or nightly) updates do check, if any new versions are available and, if so, trigger the build pipeline in the develop branch. That builds the site and runs all the tests. When all tests pass successfully, as release for the Drupal site is generated by merging into main where the build and test pipelines run again.

For development, that takes longer before it should be released, feature branches are used on the level of the Drupal site project, where temporarily dependencies can be on development branches of components as well. Those feature branches can be built, tested and deployed just like die default branches as well, without any impact on them whatsoever.

The problem with that current structure

There are 2 problems with this setup:

1. The pipelines run nightly, and if successful, the same tests run twice after each other; first on develop and afterwards on main. Both pipeline work with the same data and code base. So, there is no point in running the tests twice.
2. If anything that the team works on during the day, breaks the overall site, this is only recognized at night, i.e. very late. Nobody is there to fix that until the next day.

Proposed changes

The workflow with develop, main and feature branches has one important assumption: the develop branch only ever contains code which is deployable to the live site.

With that in mind, the following changes should be introduced:

• The Check for updates pipeline on develop runs much more often, i.e. every 30 minutes or once per hour. This will help to recognize possible issues much earlier because the tests in the pipeline will show within a short period of time, and the developer can then react to such problems and fix them right away.
• If the develop pipeline succeeded, it will no longer create a release of the main branch.
• For Drupal sites with automatic updated, there can still be a nightly pipeline which checks for changes in the develop branch and if the latest pipeline tests ran without failures. If so, it merges develop into main, builds and deploys the site but won't run the tests again because they would just be the same as the previous tests on the develop branch.

Changes to the developer workflow

When working on custom modules, where the changes should be published and deployed asap, the changes should not only be committed and pushed, but also a release for that custom module should be published so that the test process on the develop branch gets triggered. That way, the team will learn about possible new issues quickly.