cert-manager: Automatically provision TLS certificates in Kubernetes

cert-manager is an addon for automatically generating TLS certificates from Let's Encrypt for your Kubernetes cluster, which also is the official successor of kube-lego.

ref:
https://github.com/jetstack/cert-manager
https://letsencrypt.org/

If you are interfering with kube-lego, see the following link:

kube-lego: Automatically provision TLS certificates in Kubernetes
https://vinta.ws/code/kube-lego-automatically-provision-tls-certificates-in-kubernetes.html

Install

Assuming you already have Helm setup. If not, see the following link:

Helm: the package manager for Kubernetes
https://vinta.ws/code/helm-the-package-manager-for-kubernetes.html

$ helm install \
--name cert-manager \
--set rbac.create=false \
stable/cert-manager

$ helm ls --all cert-manager

$ kubectl logs deploy/cert-manager-cert-manager cert-manager -f
$ kubectl logs deploy/cert-manager-cert-manager ingress-shim -f

ref:
https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/deploying.md
https://docs.helm.sh/helm/#helm-install

Create Cluster Issuers

An Issuer is a Certificate Authority who provisions TLS Certificates for your domains, for instance, Let's Encrypt.

spec.acme.privateKeySecretRef is the Secret used to store the ACME account private key, cert-manager creates it for you.

# cert-manager/issuer.yaml
kind: ClusterIssuer
apiVersion: certmanager.k8s.io/v1alpha1
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v01.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-private-key
    http01: {}
---
kind: ClusterIssuer
apiVersion: certmanager.k8s.io/v1alpha1
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging-private-key
    http01: {}
$ kubectl apply -f cert-manager/issuer.yaml

$ kubectl get clusterissuers
$ kubectl describe clusterissuer letsencrypt-staging

$ kubectl get secrets --all-namespaces
NAMESPACE     NAME                                    TYPE                                  DATA      AGE
default       cert-manager-cert-manager-token-5j4gw   kubernetes.io/service-account-token   3         6m
kube-system   letsencrypt-prod-private-key            Opaque                                1         40s
kube-system   letsencrypt-staging-private-key         Opaque                                1         40s
...

ref:
https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/cluster-issuers.md
https://github.com/jetstack/cert-manager/tree/master/docs/api-types/issuer

Create the Ingress

Assuming you already have an Ingress like this:

# ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: simple-project
  annotations:
    kubernetes.io/ingress.class: "gce"
spec:
  rules:
  - host: kittenphile.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: simple-frontend
          servicePort: http
  - host: api.kittenphile.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: simple-api
          servicePort: http

Before you test certificate provisions, you must add A DNS records which point to the "Address" of the Ingress for all your domains.

$ kubectl apply -f ingress.yaml

$ kubectl describe ing simple-project
Name:             simple-project
Namespace:        default
Address:          12.34.56.78
Default backend:  default-http-backend:80 (10.44.2.5:8080)

$ dig kittenphile.com

Create a Staging Certificate

Let's Encrypt production API has a rate limit of 20 requests per domain per week, so it is strongly recommended to first use staging API for testing your configurations.

A Certificate contains the information required to make a certificate signing request for a given Issuer.

# cert-manager/certificate.yaml
kind: Certificate
apiVersion: certmanager.k8s.io/v1alpha1
metadata:
  name: kittenphile-com
spec:
  secretName: kittenphile-com-tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: kittenphile.com
  dnsNames:
  - kittenphile.com
  - api.kittenphile.com
  acme:
    config:
    - http01:
        ingress: simple-project
      domains:
      - kittenphile.com
      - api.kittenphile.com

ref:
https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/acme-http-validation.md
https://blog.n1analytics.com/free-automated-tls-certificates-on-k8s/

Configure the Ingress

Add domains you want to have TLS certificates to spec.tls.hosts.

spec.tls.secretName is the Secret used to store the certificate received from Let's Encrypt, i.e., tls.key and tls.crt.

# ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: simple-project
  annotations:
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
  - secretName: kittenphile-com-tls
    hosts:
    - kittenphile.com
    - api.kittenphile.com
  rules:
  - host: kittenphile.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: simple-frontend
          servicePort: http
  - host: api.kittenphile.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: simple-api
          servicePort: http

ref:
https://kubernetes.io/docs/concepts/services-networking/ingress/#tls

cert-manager watches new domain entries in any Certificate resource, requests certificates from Let's Encrypt for new domains, and creates ACME HTTP-01 challenge endpoints which are attached to the Ingress automatically.

You could see the issuing progress in "Events" section of kittenphile-com certificate.

$ kubectl logs deploy/cert-manager-cert-manager cert-manager -f

$ kubectl apply -f ingress.yaml
$ kubectl apply -f cert-manager/certificate.yaml

$ kubectl describe certificate kittenphile-com
...
Events:
  Type    Reason               Age                From                     Message
  ----    ------               ----               ----                     -------
  Normal  PresentChallenge     5m                 cert-manager-controller  Presenting http-01 challenge for domain kittenphile.com
  Normal  PresentChallenge     5m                 cert-manager-controller  Presenting http-01 challenge for domain api.kittenphile.com
  Normal  SelfCheck            5m                 cert-manager-controller  Performing self-check for domain kittenphile.com
  Normal  SelfCheck            5m                 cert-manager-controller  Performing self-check for domain api.kittenphile.com
  Normal  ObtainAuthorization  25s                cert-manager-controller  Obtained authorization for domain kittenphile.com
  Normal  ObtainAuthorization  36s                cert-manager-controller  Obtained authorization for domain api.kittenphile.com
  Normal  RenewalScheduled     19s (x3 over 23s)  cert-manager-controller  Certificate scheduled for renewal in 1438 hours
  Normal  CeritifcateIssued    19s (x3 over 24s)  cert-manager-controller  Certificated issued successfully
...

You could also find the exact ACME challenge path by inspecting your Ingress resource.

$ kubectl describe ing simple-project
...
TLS:
  kittenphile-com-tls terminates kittenphile.com,api.kittenphile.com
Rules:
  Host                Path  Backends
  ----                ----  --------
kittenphile.com
                      /*                                                  simple-frontend:http (<none>)
                      /.well-known/acme-challenge/ltvlVWEXTup5BqEsztirs   cm-kittenphile-com-gikjk:8089 (<none>)
api.kittenphile.com
                      /*                                                  simple-api:http (<none>)
                      /.well-known/acme-challenge/kd08LK93Fkdf653h9dfjj   cm-kittenphile-com-hgdkd:8090 (<none>)
...

It's also worth noting, when using the Google Cloud's Ingress controller (kubernetes.io/ingress.class: "gce"), changes to load balancers might take up to 10 minutes to propagate. cert-manager sets a timeout of 15 minutes on HTTP validations to allow for this.

ref:
https://github.com/jetstack/cert-manager/issues/285

Create a Production Certificate

After you make sure all configurations are correct, just change the Certificate manifest's spec.issuerRef.name to letsencrypt-prod. Also, delete the staging Certificate and TLS Secret.

$ kubectl delete certificate kittenphile-com && \
  kubectl delete secret kittenphile-com-tls

$ kubectl apply -f cert-manager/certificate.yaml

$ kubectl describe certificate kittenphile-com
$ kubectl describe ing simple-project

cert-manager attaches temporarily generated Services to the Ingress for presenting ACME HTTP-01 challenges of each domains, which changes configurations of the Ingress. Don't forget that Google Cloud's Ingress controller might take a long time to propagate settings.

Provision automatically with ingress-shim

As of cert-manager v0.2.4, the ingress-shim seems to have some issues, for instance, it can not detect new domains which were added after the first issuing. The workaround is to create Certificate manifests manually, in other words, don't use ingress-shim.

$ helm upgrade \
cert-manager \
stable/cert-manager \
--set ingressShim.extraArgs='{--default-issuer-name=letsencrypt-prod,--default-issuer-kind=ClusterIssuer}'

ref:
https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/ingress-shim.md

Migrate from kube-lego

Scale down and make sure kube-lego Pods are no longer running.

$ kubectl scale \
--namespace kube-lego \
--replicas=0 \
deployment kube-lego

$ kubectl get pods --namespace kube-lego

Download a copy of your ACME account private key which created by kube-lego.

$ kubectl get secret \
--namespace kube-lego \
-o yaml \
--export kube-lego-account > cert-manager/secret.yaml

Change metadata.name to something more relevant to cert-manager.

# cert-manager/secret.yaml
kind: Secret
apiVersion: v1
metadata:
  name: letsencrypt-prod-private-key
type: Opaque
data:
  acme-registration-url: XXX
  tls.key: XXX

Deploy cert-manager's Issuers and Certificates. Make sure your Certificate matches domains specified in the Ingress.

$ kubectl apply -f cert-manager/secret.yaml && \
  kubectl apply -f cert-manager/issuer.yaml && \
  kubectl apply -f cert-manager/certificate.yaml

ref:
https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/migrating-from-kube-lego.md