Deploying GitLab Runners in K8s With Self Signed Certs

- 3 mins read

tl;dr

Don’t do it, if your certificate does not have a root CA certificate attached git will not read the certificate as valid but the GitLab runner will. Your runner will authenticate but git pulls will continue to fail. If your runner is from an internal non-TLS endpoint this does not impact you.

Summary

If you have deployed GitLab via Kustomize and only have access to a self signed certificate, you will need to pass the self signed certificate into the GitLab runner allowing it to authenticate with GitLab.

Collect GitLabs certificate

You can collect the certificate easily via openssl

openssl s_client -showcerts -connect gitlab.lan:443 -servername gitlab.lan < /dev/null 2>/dev/null | openssl x509 -outform PEM

Simply replace gitlab.lan with your GitLab instance host. This will output your GitLab instance certificate.

Pass certificate to GitLab Runner

Now create a ConfigMap, I generally place all manifest types in a single file, but whatever works. Create your ConfigMap like so:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: self-signed-cert
  namespace: gitlab
  labels:
    app: runner    
data:
  gitlab.lan.crt: |
    -----BEGIN CERTIFICATE-----
    <snip>
    -----END CERTIFICATE-----

Note the gitlab.lan.crt will be the name of the file that will be placed in the runner pod. The name is important so change gitlab.lan.crt to match the hostname of your instance.

Now find your runner instance, the ConfigMap will need to be mounted at /home/gitlab-runner/.gitlab-runner/certs, like so:

spec:
  template:
    spec:
      containers:
        - name: gitlab-gitlab-runner
          volumeMounts:
            - name: self-signed-cert
              mountPath: /home/gitlab-runner/.gitlab-runner/certs
      volumes:
        - name: self-signed-cert
          configMap:
            name: self-signed-cert

This will allow your runners to validate the self signed certificate and join the GitLab server.

NOTE: If you are running GitLab and your certificate is NOT signed with a root CA your runner will still fail to run jobs. You will see the following error:

Preparing the "kubernetes" executor 00:00
Using Kubernetes namespace: gitlab
Using Kubernetes executor with image jdkato/vale:v2.24.0 ...
Using attach strategy to execute scripts...
Preparing environment 00:03
Waiting for pod gitlab/runner-gnnusbml-project-70-concurrent-0zc55d to be running, status is Pending
Running on runner-gnnusbml-project-70-concurrent-0zc55d via gitlab-gitlab-runner-5b6fffd796-n6mqx...
Getting source from Git repository 00:01
Fetching changes with git depth set to 20...
Initialized empty Git repository in /tmp/lab/hugo-site/.git/
Created fresh repository.
fatal: unable to access 'https://gitlab.lan/lab/hugo-site.git/': SSL: couldn't get X509-issuer name!
Cleaning up project directory and file based variables

The runner will allow an empty root CA however git will not allow this. You have two options at this point, deploy a CA in cert-manager and cut a new certificate for GitLab OR disable SSL verify for git. The later can be achieved by adding the following to your GitLab CI file:

variables:
  GIT_SSL_NO_VERIFY: "true"

This is not a good option in general, as a result a post will be coming out explaining how to deploy a private CA to cert-manager.

Sources

Update

An update on how to deploy your own external CA: Deploying External Private CA to cert-manager