Re-factor download and merge scripts to use GLab CLI

Hello,

we have a specific problem that prevents us to use a current pull-sql and similar commands.

Namely, our GitLab URL is https://gitlab.company.net, but our URLs to the repo are:

SSH: git@git.company.net:group/project.git HTTPS: https://gitlab.company.net/group/project.git (I have no idea how it's possible, where is it configured, etc)

so it fails here: https://gitlab.lakedrops.com/docker/gitlab-drupal-ci/-/blob/main/bin/download#L82

as git.company.net is not equal to gitlab.company.net.

I have tried to use https://git.company.net as GitLab URL, but it didn't work, as API is located at https://gitlab.company.net/api/v4/projects...

So I would say we'll need some safer method for this check. I did some more research, and noticed this in the API response:

[
  {
    "id": 1,
    "description": null,
    "name": "Project",
    "name_with_namespace": "Group / Project",
    "path": "group-project",
    "path_with_namespace": "group/project",
    "created_at": "2022-09-30T13:46:02Z",
    "default_branch": "main",
    "tag_list": [
      "example",
      "project"
    ],
    "topics": [
      "example",
      "project"
    ],
    "ssh_url_to_repo": "git@git.example.net:group/project.git",
    "http_url_to_repo": "https://gitlab.example.net/group/project.git",
    "web_url": "https://gitlab.example.net/group/project",
...

so can we use ssh_url_to_repo and http_url_to_repo values to check if the current project belongs there...? If so, I can do some adjustments to the /bin/download bash script & try to make it work.

Or do you think there is a better idea/solution?

changed the description

Nice one :-)

I think we should first review all Docker and Composer projects to see where else we use the GITLAB_URL environment variable. Because if such an issue happens here, it could well happen elsewhere too.

Once we know the full scope of the issue, we can discuss what the best resolution would be for that.

I'll have a look and come back to this later today.

@jurgenhaas thank you!

https://stackoverflow.com/questions/22458344/change-ssh-host-for-cloning - probably relevant, how to change ssh clone host in GitLab

Well, that would require a config change on your GitLab instance. Not sure, if you want to do that since it could easily break other stuff.

And also, this might be an issue for other users in the future as well, so we'd rather fix it now.

OK, the good news is that the scope remains small. The GITLAB_URL is only used in this docker project, but in 2 files: download and merge. That reminds me that those 2 files contain a lot of redundant code and are due for a refactoring anyways. I wonder if now is the right time to fix that. And if so, should we rewrite those 2 script files entirely and use the GitLab CLI instead? It is already part of this package (see https://gitlab.com/gitlab-org/cli) and it would be much easier to do certain things with that rather than using bash and curl.

It depends a bit how quickly you need that fix. If you need it today, it can't be done with a proper approach, then only a quick fix will help. If we have a bit more time like a few days, then I'd prefer the proper approach.

added To Do label

@jurgenhaas great, thank you! CLI looks interesting, I didn't know before about that!

It depends a bit how quickly you need that fix. If you need it today, it can't be done with a proper approach, then only a quick fix will help. If we have a bit more time like a few days, then I'd prefer the proper approach.

It's not urgent at all! I already have a workaround, but also we can use chatops (mattermost) commands to get DB dumps/files, etc.

That's great news. So, I'll get to re-factor those script asap and let you know when they are ready for testing.

changed title from "This repository doesn't match the provided GitLab instance" to Re-factor download and merge scripts to use GLab CLI

assigned to @jurgenhaas

added Doing label and removed To Do label

mentioned in commit gitlab-ci-cd/drupal@b2f9638c

mentioned in commit 4ea53c33

mentioned in commit 3e4bc840

mentioned in commit f7c74b4d

mentioned in merge request !37 (merged)

mentioned in commit l3d@d0bd6c52

This is now implemented with the GitLab CLI client, and the code is much cleaner and much more reliable. The environment variables GITLAB_URL and GITLAB_PRIVATE_TOKEN are no longer required. The client grabs all required information from the current git repository.

To use the client, you need to authenticate once on your host. Use glab auth login --help to see the available options. The authentication will be made persistent in your ~/.config/glab-cli/config.yml file and is available in all L3D projects. The client also allows working with any number of GitLab instances, not only one.

/cc @danielspeicher /cc @richardpapp

One little caveat though: when using the download script, there is a glab command to wait for the pipeline to finish, which pulls the database dump from a server. This command requires a user confirmation at the end, but I guess this is well worth it.

@dejan-acolono this is available in L3D version 2.8.1 - please give it a try.

closed

removed Doing label

mentioned in commit gitlab-ci-cd/general@ea451394

@jurgenhaas looks awesome, thank you!

but in our case, I think we still have the same problem :/

Trigger pipeline ... none of the git remotes configured for this repository points to a known GitLab host. Please use glab auth login to authenticate and configure a new host for glab

but let me try & re-configure glab authentication somehow

okay, I think I have that part working:

glab auth login
What GitLab instance do you want to log into? GitLab Self-hosted Instance
GitLab hostname: git.company.net
API hostname: gitlab.company.net
...

but on a current pull-sql now I am getting:

POST https://gitlab.company.net/api/v4/projects/group/project/pipeline: 400 {message: {base: [Pipeline will not run for the selected trigger. The rules configuration prevented any jobs from being added to the pipeline.]}}

let me see what can be wrong with that

I've had that issue when the pipeline didn't produce any jobs because of rules and conditions. Which pipeline configuration do you use, the one from lakedrops or your own?

@jurgenhaas so, I cloned your https://gitlab.lakedrops.com/gitlab-ci-cd/drupal project into our Gitlab some weeks ago, as I was unable to make use of it remotely. I mean, I can use

include: remote: 'https://gitlab.com/awesome-project/raw/main/.before-script-template.yml'

but then any script included in .before-script-template.yml (on your side) doesn't work for us (script not found or something like that). Do you have maybe any solution/workaround for that?

But yes, our gitlab-ci-cd/drupal probably is not up to date with yours. Let me check it

You should be able to use our template by simply using this in your .gitlab-ci.yml:

include:
  - project: 'gitlab-ci-cd/drupal'
    ref: main
    file: '/lakedrops.yml'

and modify that to use it from the remote server. Then you don't need to include the before-script separately, do you? But that's probably a separate issue on how to use the templates from a remote host.

@jurgenhaas yes, exactly, that's another problem I've spent already hours on, but didn't manage to make it working

so I have just replaced

include:
  - project: 'gitlab-ci-cd/drupal'
    ref: main
    file: '/lakedrops.yml'

with

include:
  remote: 'https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/raw/main/lakedrops.yml'

and pipeline fails:

Unable to create pipeline
Project `gitlab-ci-cd/drupal` not found or access denied! Make sure any includes in the pipeline configuration are correctly defined.

and my understanding is that it was not able to determine correctly includes like this: https://gitlab.lakedrops.com/gitlab-ci-cd/drupal/-/blob/main/lakedrops.yml#L42

I found this issue: https://gitlab.com/gitlab-org/gitlab/-/issues/35180 but not sure if it still is not possible todo

Hmm, interesting. But that's a different issue then. Let's have a look at this in one of the template projects.

For here, you may try the download script by just updating the templates into your fork, then it should work.

One note of caution though: the glab client does not exist in the PHP 7.4 version of L3D, I wasn't able to install it there yet.

@dejan-acolono just managed to also get the glab client into the PHP 7.4 container. This is available in L3D version 2.8.2

For here, you may try the download script by just updating the templates into your fork, then it should work.

I can confirm that a current pull-sql now worked :) Thanks once again @jurgenhaas

That's amazing !!!

mentioned in commit b0d92e29

mentioned in commit gitlab-ci-cd/drupal@87fb8aaa

Hello Team,

I am facing now pipeline related issue during 'a current pull-sql' on Mac M1.

Please check attached screen shot.

@slsawhney thanks for reporting this issue. Note, this was closed about a month ago and commenting on closed issues is not a good idea. Chances are that this gets missed. Starting a new issue then is recommended. Just a tip for next time.

As for your issue, there are several questions:

• Does the glab client recognize the project URL correctly? Is the printed URL in your screenshot identifying the location of the project correctly?
• Have you authenticated your glab client to talk to your GitLab? This would usually work with glab auth login (required once on your host) and you can verify the status with glab auth status

reopened

Hi @slsawhney any chance to get some feedback on this? Would really like to help resolving this, but that needs more input.

unassigned @jurgenhaas

Hello @jurgenhaas,

Today I have re-setup the project. Everything worked perfectly.

Thanks for your time.

Hi @slsawhney that's good news, thanks a lot for your feedback.

closed