cert-manager: Automatically provision TLS certificates in Kubernetes

cert-manager: Automatically provision TLS certificates in Kubernetes

cert-manager is an addon for automatically generating TLS certificates from Let's Encrypt for your Kubernetes cluster, which also is the official successor of kube-lego.

ref:
https://github.com/jetstack/cert-manager
https://letsencrypt.org/

If you are interfering with kube-lego, see the following link:

kube-lego: Automatically provision TLS certificates in Kubernetes
https://vinta.ws/code/kube-lego-automatically-provision-tls-certificates-in-kubernetes.html

Install

Assuming you already have Helm setup. If not, see the following link:

Helm: the package manager for Kubernetes
https://vinta.ws/code/helm-the-package-manager-for-kubernetes.html

$ helm install \
--name cert-manager \
--set rbac.create=false \
stable/cert-manager

$ helm ls --all cert-manager

$ kubectl logs deploy/cert-manager-cert-manager cert-manager -f
$ kubectl logs deploy/cert-manager-cert-manager ingress-shim -f

ref:
https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/deploying.md
https://docs.helm.sh/helm/#helm-install

Create Cluster Issuers

An Issuer is a Certificate Authority who provisions TLS Certificates for your domains, for instance, Let's Encrypt.

spec.acme.privateKeySecretRef is the Secret used to store the ACME account private key, cert-manager creates it for you.

# cert-manager/issuer.yaml
kind: ClusterIssuer
apiVersion: certmanager.k8s.io/v1alpha1
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v01.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-prod-private-key
    http01: {}
---
kind: ClusterIssuer
apiVersion: certmanager.k8s.io/v1alpha1
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging.api.letsencrypt.org/directory
    email: [email protected]
    privateKeySecretRef:
      name: letsencrypt-staging-private-key
    http01: {}
$ kubectl apply -f cert-manager/issuer.yaml

$ kubectl get clusterissuers
$ kubectl describe clusterissuer letsencrypt-staging

$ kubectl get secrets --all-namespaces
NAMESPACE     NAME                                    TYPE                                  DATA      AGE
default       cert-manager-cert-manager-token-5j4gw   kubernetes.io/service-account-token   3         6m
kube-system   letsencrypt-prod-private-key            Opaque                                1         40s
kube-system   letsencrypt-staging-private-key         Opaque                                1         40s
...

ref:
https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/cluster-issuers.md
https://github.com/jetstack/cert-manager/tree/master/docs/api-types/issuer

Create the Ingress

Assuming you already have an Ingress like this:

# ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: simple-project
  annotations:
    kubernetes.io/ingress.class: "gce"
spec:
  rules:
  - host: kittenphile.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: simple-frontend
          servicePort: http
  - host: api.kittenphile.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: simple-api
          servicePort: http

Before you test certificate provisions, you must add A DNS records which point to the "Address" of the Ingress for all your domains.

$ kubectl apply -f ingress.yaml

$ kubectl describe ing simple-project
Name:             simple-project
Namespace:        default
Address:          12.34.56.78
Default backend:  default-http-backend:80 (10.44.2.5:8080)

$ dig kittenphile.com

Create a Staging Certificate

Let's Encrypt production API has a rate limit of 20 requests per domain per week, so it is strongly recommended to first use staging API for testing your configurations.

A Certificate contains the information required to make a certificate signing request for a given Issuer.

# cert-manager/certificate.yaml
kind: Certificate
apiVersion: certmanager.k8s.io/v1alpha1
metadata:
  name: kittenphile-com
spec:
  secretName: kittenphile-com-tls
  issuerRef:
    name: letsencrypt-staging
    kind: ClusterIssuer
  commonName: kittenphile.com
  dnsNames:
  - kittenphile.com
  - api.kittenphile.com
  acme:
    config:
    - http01:
        ingress: simple-project
      domains:
      - kittenphile.com
      - api.kittenphile.com

ref:
https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/acme-http-validation.md
https://blog.n1analytics.com/free-automated-tls-certificates-on-k8s/

Configure the Ingress

Add domains you want to have TLS certificates to spec.tls.hosts.

spec.tls.secretName is the Secret used to store the certificate received from Let's Encrypt, i.e., tls.key and tls.crt.

# ingress.yaml
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: simple-project
  annotations:
    kubernetes.io/ingress.class: "gce"
spec:
  tls:
  - secretName: kittenphile-com-tls
    hosts:
    - kittenphile.com
    - api.kittenphile.com
  rules:
  - host: kittenphile.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: simple-frontend
          servicePort: http
  - host: api.kittenphile.com
    http:
      paths:
      - path: /*
        backend:
          serviceName: simple-api
          servicePort: http

ref:
https://kubernetes.io/docs/concepts/services-networking/ingress/#tls

cert-manager watches new domain entries in any Certificate resource, requests certificates from Let's Encrypt for new domains, and creates ACME HTTP-01 challenge endpoints which are attached to the Ingress automatically.

You could see the issuing progress in "Events" section of kittenphile-com certificate.

$ kubectl logs deploy/cert-manager-cert-manager cert-manager -f

$ kubectl apply -f ingress.yaml
$ kubectl apply -f cert-manager/certificate.yaml

$ kubectl describe certificate kittenphile-com
...
Events:
  Type    Reason               Age                From                     Message
  ----    ------               ----               ----                     -------
  Normal  PresentChallenge     5m                 cert-manager-controller  Presenting http-01 challenge for domain kittenphile.com
  Normal  PresentChallenge     5m                 cert-manager-controller  Presenting http-01 challenge for domain api.kittenphile.com
  Normal  SelfCheck            5m                 cert-manager-controller  Performing self-check for domain kittenphile.com
  Normal  SelfCheck            5m                 cert-manager-controller  Performing self-check for domain api.kittenphile.com
  Normal  ObtainAuthorization  25s                cert-manager-controller  Obtained authorization for domain kittenphile.com
  Normal  ObtainAuthorization  36s                cert-manager-controller  Obtained authorization for domain api.kittenphile.com
  Normal  RenewalScheduled     19s (x3 over 23s)  cert-manager-controller  Certificate scheduled for renewal in 1438 hours
  Normal  CeritifcateIssued    19s (x3 over 24s)  cert-manager-controller  Certificated issued successfully
...

You could also find the exact ACME challenge path by inspecting your Ingress resource.

$ kubectl describe ing simple-project
...
TLS:
  kittenphile-com-tls terminates kittenphile.com,api.kittenphile.com
Rules:
  Host                Path  Backends
  ----                ----  --------
kittenphile.com
                      /*                                                  simple-frontend:http (<none>)
                      /.well-known/acme-challenge/ltvlVWEXTup5BqEsztirs   cm-kittenphile-com-gikjk:8089 (<none>)
api.kittenphile.com
                      /*                                                  simple-api:http (<none>)
                      /.well-known/acme-challenge/kd08LK93Fkdf653h9dfjj   cm-kittenphile-com-hgdkd:8090 (<none>)
...

It's also worth noting, when using the Google Cloud's Ingress controller (kubernetes.io/ingress.class: "gce"), changes to load balancers might take up to 10 minutes to propagate. cert-manager sets a timeout of 15 minutes on HTTP validations to allow for this.

ref:
https://github.com/jetstack/cert-manager/issues/285

Create a Production Certificate

After you make sure all configurations are correct, just change the Certificate manifest's spec.issuerRef.name to letsencrypt-prod. Also, delete the staging Certificate and TLS Secret.

$ kubectl delete certificate kittenphile-com && \
  kubectl delete secret kittenphile-com-tls

$ kubectl apply -f cert-manager/certificate.yaml

$ kubectl describe certificate kittenphile-com
$ kubectl describe ing simple-project

cert-manager attaches temporarily generated Services to the Ingress for presenting ACME HTTP-01 challenges of each domains, which changes configurations of the Ingress. Don't forget that Google Cloud's Ingress controller might take a long time to propagate settings.

Provision automatically with ingress-shim

As of cert-manager v0.2.4, the ingress-shim seems to have some issues, for instance, it can not detect new domains which were added after the first issuing. The workaround is to create Certificate manifests manually, in other words, don't use ingress-shim.

$ helm upgrade \
cert-manager \
stable/cert-manager \
--set ingressShim.extraArgs='{--default-issuer-name=letsencrypt-prod,--default-issuer-kind=ClusterIssuer}'

ref:
https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/ingress-shim.md

Migrate from kube-lego

Scale down and make sure kube-lego Pods are no longer running.

$ kubectl scale \
--namespace kube-lego \
--replicas=0 \
deployment kube-lego

$ kubectl get pods --namespace kube-lego

Download a copy of your ACME account private key which created by kube-lego.

$ kubectl get secret \
--namespace kube-lego \
-o yaml \
--export kube-lego-account > cert-manager/secret.yaml

Change metadata.name to something more relevant to cert-manager.

# cert-manager/secret.yaml
kind: Secret
apiVersion: v1
metadata:
  name: letsencrypt-prod-private-key
type: Opaque
data:
  acme-registration-url: XXX
  tls.key: XXX

Deploy cert-manager's Issuers and Certificates. Make sure your Certificate matches domains specified in the Ingress.

$ kubectl apply -f cert-manager/secret.yaml && \
  kubectl apply -f cert-manager/issuer.yaml && \
  kubectl apply -f cert-manager/certificate.yaml

ref:
https://github.com/jetstack/cert-manager/blob/master/docs/user-guides/migrating-from-kube-lego.md

Helm: the package manager for Kubernetes

Helm: the package manager for Kubernetes

Helm is a tool for managing Kubernetes charts. Charts are packages of pre-configured Kubernetes resources. Think of it like apt for Kubernetes.

ref:
https://github.com/kubernetes/helm

Install

Install the Helm client, helm.

$ wget https://kubernetes-helm.storage.googleapis.com/helm-v2.8.2-darwin-amd64.tar.gz && \
tar -zxvf helm-v2.8.2-darwin-amd64.tar.gz && \
sudo mv darwin-amd64/helm /usr/local/bin

# or

$ brew install kubernetes-helm

ref:
https://docs.helm.sh/using_helm/#installing-helm

Install the Helm server, tiller, which might take several minutes.

You could simply deploy tiller if you don't have RBAC.

$ helm init
$ kubectl get all --namespace kube-system

# uninstall tiller
$ helm reset

Otherwise, you must create a ClusterRoleBinding which specifies a ClusterRole and a ServiceAccount if your cluster enables RBAC.

# helm/rbac.yaml
kind: ServiceAccount
apiVersion: v1
metadata:
  name: tiller
  namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
  name: tiller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
  - kind: ServiceAccount
    name: tiller
    namespace: kube-system
$ kubectl apply -f helm/rbac.yaml && \
  helm init --service-account tiller

ref:
https://docs.helm.sh/using_helm/#installing-tiller

Usage

# show both the client and server version to make sure installation is correct
$ helm version
Client: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.8.2", GitCommit:"a80231648a1473929271764b920a8e346f6de844", GitTreeState:"clean"}

# install a Chart without RBAC
$ helm install \
--name cert-manager \
--set rbac.create=false \
stable/cert-manager

# list Releases
$ helm ls
$ helm ls --all cert-manager

# delete a Release
$ helm del --purge cert-manager

ref:
https://docs.helm.sh/helm/#helm

Charts

ref:
https://github.com/kubernetes/charts

Find circular imports in Python

Find circular imports in Python

What is circular imports?
http://stackabuse.com/python-circular-imports/

You could use python -vv to inspect import relations.

$ python -vv manage.py shell
>>> from api.models import Application
>>> from member.views.site import signup

or

$ python -vv
>>> import os
>>> os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'streetvoice.settings')
>>> import django
>>> django.setup()
>>> from api.models import Application

ref:
https://stackoverflow.com/questions/6351805/cyclic-module-dependencies-and-relative-imports-in-python
https://stackoverflow.com/questions/9098787/tool-for-pinpointing-circular-imports-in-python-django

Python 3.7 has new feature to show time for importing modules. This feature is enabled with -X importtime option or PYTHONPROFILEIMPORTTIME=1 environment variable.

$ python3.7 -X importtime -c "import pipenv"

ref:
https://dev.to/methane/how-to-speed-up-python-application-startup-time-nkf

碼天狗週刊 第 120 期 @vinta - Python, Docker, Kubernetes, Code Review

碼天狗週刊 第 120 期 @vinta - Python, Docker, Kubernetes, Code Review

本文同步發表於 CodeTengu Weekly - Issue 120

Remotely debug a Python app inside a Docker container in Visual Studio Code

之前因為開發環境都 containerized 了,但是偶爾又想要用一下 Visual Studio Code 的 Python debugger,所以就試了一下 Remote Debugging 功能,順便寫了一篇網誌記錄了跟 Flask 的 --reload 併用的奇技淫巧。忍不住抱怨一下,VS Code 的 Python debugger 雖然很炫炮,尤其是 "debug.inlineValues": true,但是啟動速度真的慢到靠北啊。

不過就像 @WanCW‏ 說的,與其一直花時間追求開發環境和正式環境的一致,整天搞那些 configs,還不如用那些時間多寫幾個 test cases 啊。

P.S. 這裡的 remote 主要指的是本機的某個 Docker container,當然也可以真的是遠端的某台機器。

kennethreitz/setup.py: A Human's Ultimate Guide to setup.py

也是出自 @kennethreitz 大神之手,不過這次不是什麼改變 Python 生態系的新專案,而是一個 setup.py 的範例,有自己的 Python package 的人可以參考看看。看了之後才知道,原來有 cmdclass 這麼一個用法啊。

Kubernetes Deconstructed: Understanding Kubernetes by Breaking It Down

在你讀完八百多頁的 Kubernetes 文件之後,這個影片可能會是個很好的複習。

Kubernetes NodePort vs LoadBalancer vs Ingress? When should I use what?

看了半天的 Kubernetes 官方文件,結果反而是在讀了這篇文章之後才終於搞清楚 ServiceIngress 的使用場景,嘖。

Getting started with Data Engineering

這篇文章概略地提到了作為一個 Data Engineer 可能會需要碰到的各項技術,一個人要完全掌握這些東西當然是不太可能啦,但是看了那個 Big Data Landscape 的圖還是覺得很血汗啊,大家感受一下。

不過讀完文章之後,最大的心得反而是所謂的 Data Engineer 做的事情跟 Web Backend Developer 其實很像啊,都是圍繞著 Data 本身打轉,在技術上也有相當大的重疊,最大的差別可能是在於人家用的那套 framework 叫做 Apache Spark,逼搞夏趴。

How to Do Code Reviews Like a Human (Part One)

雖然 code review 已經是老生常談了,但是這篇文章寫得真的是好。無論你是那個覺得 code review 是件苦差事的 reviewer 或是老是被要求改 code 改到往心裡去的 reviewee,又或者你們團隊根本還沒開始做 code review,本文都非常值得一讀。

延伸閱讀:

YouGlish - Use YouTube to improve your English pronunciation

因為公司的 System Architect 是加拿大人,所以平常跟他在開會或閒聊都是全英文進行,對土台灣人如我多少還是有點不習慣,最近的困擾是關於某些單字的唸法,一般的英文單字也就算了,畢竟網路上隨便一個字典都有真人發音。但是那些我們平常在用的技術詞彙(例如 framework 或 library 的名字)就不一定在字典裡找得到了。正好前陣子 @SammyLinTw 推薦了 YouGlish 這個網站,它可以直接幫你列出 YouTube 上關於某個單字的發音片段,實用!

大家感受一下:

延伸閱讀:

瑞克和莫蒂:我酗酒、我变态,但我是个好外公

(該文有大量劇透)

在 Twitter 上看到有人分享了 dannvix/NflxMultiSubs,幫 Netflix 加上「雙語字幕」功能的瀏覽器外掛。突然就想到前一陣子因為椎間盤突出在家裡休養的那段日子,除了物理治療和皮拉提斯之外,其他時間好像都是在看 Netflix 啊。說到這裡,忍不住要跟大家分享一下我印象最深的一部作品,Rick and Morty。不誇張,這部真的是我這輩子看過最屌的科幻動畫,有點像是把銀河便車指南裡的英式幽默用 Futurama 和辛普森家庭取代的感覺,第一次看就看傻了我。非常推薦!每一集的片尾都有一段彩蛋,千萬不要錯過了。

不過如果你真的不喜歡動畫,那也可以看 Dirk Gently's Holistic Detective Agency,原著小說的作者正是 Douglas Adams。

@vinta 分享。

sindresorhus/awesome-scifi: Sci-Fi worth consuming

科幻迷大放送!

延伸閱讀:

@vinta 分享。