Deploying External Private CA to cert-manager

- 5 mins read

Summary

As a result of migrating from a single cluster with external access, to multiple internal clusters, cert-manager (on some clusters) necessarily lost access to Let’s Encrypt. This has lead to self signed certificates being used on all (intra-cluster) HTTPS endpoints internally. Realistically this is fine, however an OPNsense firewall is available and this gives the option to use its Trust function. In this example an intermediate certificate authority will be created for one of the three clusters (all three will get their own, but that does not need to be shown here).

Application versions

  • OPNsense: 23.7.5
    • OpenSSL: 1.1.1w 11 Sep 2023
  • Kubernetes: v1.26.9+rke2r1
    • cert-manager: 1.13.1

Tools

All clusters in this deployment are deployed via Kustomize, this includes cert-manager. Keep this in mind for the Import Intermediate CA into cert-manager section.

Create Root CA

Creating a root certificate is very straight forward. This can be found in “System” -> “Trust” -> “Authorities” -> big plus button on top right. Change the “Method” to “Create an internal Certificate Authority” then simply fill out the fields, the image below has made some tweaks to the defaults but they are not needed. Then click save.

root-CA

Create Intermediate CA

Creating the intermediate CA is pretty much just as simple, from the same location used to create the root CA, click the big plus button on top right to create the intermediate CA. Change the “Method” to “Create an intermediate Certificate Authority”. When filling out the fields ensure the “Signing Certificate Authority” selected is the root CA from the previous step. For your own sanity set a good CN (Common Name), OU (Organizational Unit), and “Descriptive name”. intermediate-CA

Import Intermediate CA into cert-manager

OoP

  1. Prepare certificates and CA key
  2. Encode certs and keys
  3. Create CA secret
  4. Create cluster issuer

Prepare certificates and CA key

Return to the “Authorities” section (“System” -> “Trust” -> “Authorities”) and download the certificates for both your root CA, and your intermediate CA. Download only the intermediate CA’s key.

Once both CA certificates are downloaded, you will need to merge the root and intermediate and it will need to be in the correct order. This is very simple to do as there are only two CAs here. Simply copy the root certificate below the intermediate, or make life easy:

cat ./Lab-CA.crt >> ./Services+Cluster+CA.crt

Where Lab-CA is the root, and Services+Cluster+CA is the intermediate.

Encode certs and keys

As usual with Kubernetes secrets need to be base64 encoded, given both the key and certs are multi-line they can be encoded with -w 0 to disable line wrapping. In this example this can be done like so:

cat ./Services+Cluster+CA.crt| base64 -w 0
cat ./Services+Cluster+CA.key | base64 -w 0

Save the output while they are added to a new secret.

Create CA secret

Below is a secret snippet, simply paste the encoded strings in their respective spots, cert-manager will look specifically for the keys tls.crt and tls.key. Change the name and namespace if need be, then deploy the secret.

---
apiVersion: v1
kind: Secret
metadata:
    name: ca-key-pair
    namespace: cert-manager
data:
    tls.crt: 
    tls.key:

Note: This page assumes you are securely encrypting your secrets (i.e. sops with KMS, PGP, etc).

Create cluster issuer

Deploying your ClusterIssuer is just as easy, example below:

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: internal-serv-ca
spec:
  ca:
    secretName: ca-key-pair

obviously if you changed the name of the secret the secretName will need to be adjusted to match that. Due to the fact this is a ClusterIssuer there is no need to set a namespace as this is a cluster scoped resource, however, the Issuer CRD would look nearly identical:

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: internal-serv-ca
  namespace: my-app
spec:
  ca:
    secretName: ca-key-pair

If you choose to roll with a ClusterIssuer it needs to be kept in mind that a certificate can be cut from any namespace, so there are security implications to keep in mind. The infrastructure this CA is being deployed to has each cluster responsible for their own respective domain, for example:

  • Management Cluster: *.mgmt.lan
  • Applications Cluster: *.apps.lan
  • Services Cluster: *.serv.lan

A note on IngressRoute TLS certificates

When using cert-manager with lets encrypt you will get a completely standard certificate which is obviously what most user would want. Certificates requested from cert-manager with an external CA (this may also apply to internal CAs) the root CA certificate will be moved to a separate field, the end result is invalid certificates at least from Firefox and Chrome’s perspective, this can be remedied easily though. When cutting a certificate simply add “server auth” to spec.usages like below:

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: minio.serv.lan
spec:
  dnsNames:
    - minio.serv.lan
  secretName: minio.serv.lan
  issuerRef:
    name: internal-serv-ca
    kind: ClusterIssuer
  usages:
    - server auth

This will bundle the root CA and intermediate certificates together for the HTTPS endpoint certificate.

Deploying CA root to via Ansible

To make life easier you can deploy your new root CA to all Debian and RHEL system via Ansible with the following tasks. The example below was placed into an internally made/managed “generic” role, and the root certificate was placed in the roles “files” folder.

---
- name: Install root CA (RHEL)
  when: ansible_os_family == "RedHat"
  block:
    - name: Make sure the folder exists (Red Hat)
      ansible.builtin.file:
        path: /etc/pki/ca-trust/source/anchors
        state: directory

    - name: Copy PEM into folder (Red Hat)
      ansible.builtin.copy:
        src: root-CA.crt
        dest: "/etc/pki/ca-trust/source/anchors/root-CA.crt"
      register: result

    - name: Update CA Trust (Red Hat)
      ansible.builtin.command: update-ca-trust
      when: result is changed

- name: Install root CA (Debian)
  when: ansible_os_family == "Debian"
  block:
    - name: Make sure the folder exists (Debian)
      ansible.builtin.file:
        path: /usr/local/share/ca-certificates
        state: directory

    - name: Copy the certificate (Debian)
      ansible.builtin.copy:
        src: root-CA.crt
        dest: "/usr/local/share/ca-certificates/root-CA.crt"
      register: result

    - name: Update CA Trust (Debian)
      ansible.builtin.command: update-ca-certificates
      when: result is changed

Deploying to NixOS

NixOS makes life very easy, you can simply place the root certificate in your configuration.nix like so:

 security.pki.certificates = [
 ''
   -----BEGIN CERTIFICATE-----
   <snip>
   -----END CERTIFICATE-----
 ''];

Sources