碼天狗週刊 第 125 期 @vinta - Amazon Web Services, Google Cloud Platform, Kubernetes, DevOps, MySQL, Redis

碼天狗週刊 第 125 期 @vinta - Amazon Web Services, Google Cloud Platform, Kubernetes, DevOps, MySQL, Redis

本文同步發表於 CodeTengu Weekly - Issue 125

Apex and Terraform: The easiest way to manage AWS Lambda functions

因為一直都有訂閱 RSS 的習慣,但是常常工作一忙就積了一堆文章忘記看,可是又發現自己就算上班事情很多還是會三不五時刷一下 Twitter ~~順便抱怨幾句~~,所以就乾脆建了一個 @vinta_rss_bot,透過 Zapier 同步 Feedly 裡的文章到 Twitter,讓自己在刷推的時候很容易不小心就看到。實測了一個多禮拜,效果不錯,大家可以自己建一個 RSS bot 試試。

雖然這個 RSS bot 用了 Zapier 才花五分鐘就搞定了,連一行 code 都不用寫,但是因為不是每個人都是「空格之神」的信徒,一看到 @vinta_rss_bot 推了幾則沒有在標題的中英文之間加上空格的文章之後,開始覺得渾身不舒服。最後實在受不了,就用 AWS Lambda 寫了一個加空格的 web API - api.pangu.space,讓 Zapier 在輸出到 Twitter 之前先打一次。

(前情提要有點太長)

這篇文章就是紀錄我當初用 ApexTerraform 部署 AWS Lambda functions 的過程,主要的邏輯很簡單,是用 Go 寫的,比較麻煩的反而是在配置 Amazon API Gateway 和 custom domain 的 HTTPS 之類的。因為只是個 side project,所以就沒用太重量級的 Serverless 了。

延伸閱讀:

cert-manager: Automatically provision TLS certificates in Kubernetes

目前公司的 Kubernetes cluster 是用 kube-lego 自動從 Let's Encrypt 取得 TLS/SSL 憑證,但是因為 kube-lego 之前宣佈只支援到 Kubernetes v1.8 為止,所以希望大家改用另外一套由同一群人開發的在做同一件事的工具:cert-manager。

這篇文章就是紀錄我當初部署 cert-manager 的過程,準備之後從 kube-lego 遷移過去。不過因為當時測試的時候發現 cert-manager 有些功能還不是很完善,例如 ingress-shim,再加上我們在 Kubernetes v1.9.6 用 kube-lego 其實也沒遇到什麼問題,所以後來的結論是暫時先不遷移。不過文章寫都寫了,還是跟大家分享一下,希望對其他人有幫助。

延伸閱讀:

GCP products described in 4 words or less

之前都是用 AWS 比較多,但是現在公司是用 Google Cloud Platform,這篇文章可以讓你快速了解 GCP 上面有哪些東西可以用。

忍不住抱怨一下,Google Cloud Memorystore 到底什麼時候才要上線呢?

雖然 GCP 在各方面都還是差了 AWS 一截(Google Kubernetes Engine 除外),但是 Google Cloud 的 Stackdriver 系列真心好用,例如 Logging 可以直接全文搜尋所有 containers 的 stdout,什麼配置都不用(轉頭望向 ELK)。說到看 logs,kubetail 也是不錯,就是強化版的 kubectl logs -f;另外還有 Debugger 可以直接在 production code 上跑 debugger,實在炫炮。

延伸閱讀:

One Giant Leap For SQL: MySQL 8.0 Released

MySQL 8.0 前陣子發佈了,這個版本對 SQL 標準的支援有了長足的進步,終於從 SQL-92 的魔障中走出來了。有望擺脫 Friends don't let friends use MySQL 的罵名(目前看來會繼承這個污名的應該是 MongoDB)。

是說因為以前一直都在用 MySQL,根本不知道 Window functions 是什麼,第一次用 OVER (PARTITION BY ... ORDER BY ...) 反而是在 Apache Spark 裡啊(SQL 俗)。

延伸閱讀:

Redis in Action

上禮拜花了一點時間研究 Redis 的 RDB/AOF persistence 和 Master/Slave replication 的原理,發現除了官方文件之外,Redis in Action 這本書寫得也非常詳細(雖然有些內容可能有點舊了),但是畢竟是經過 Redis 作者本人背書的,值得一讀。

忍不住分享一下,我上禮拜仔細看了 Redis 4.0 的 redis.conf 之後,才發現現在多了一個 aof-use-rdb-preamble 設定,實測啟用之後可以讓 appendonly.aof 的檔案大小減少 50%,大家有空可以試試。

延伸閱讀:

金丝雀发布、滚动发布、蓝绿发布到底有什么差别?关键点是什么?

看了這篇文章我才終於知道 Canary Releases, Blue-green Deployment, Rolling Update 是什麼意思(汗顏)。

HTTP codes as Valentine’s Day comics

這篇文章用漫畫的方式介紹了各種 HTTP status code,有點太可愛了。

@vinta 分享。

Monty Python's Flying Circus on Netflix

各位觀眾,Netflix 上有 Monty Python's Flying Circus 了!不知道 Monty Python 是誰的,我們在 Issue 6 有介紹過!

@vinta 分享!

IPFS: The (Very Slow) Distributed Permanent Web

IPFS: The (Very Slow) Distributed Permanent Web

IPFS stands for InterPlanetary File System, but you could simply consider it as a distributed, permanent, but ridiculously slow, not properly functioning version of web. You could upload any static file and static website to IPFS. And the whole swarm would probably distribute your files to the moon, that might be why IPFS is so fucking slow.

ref:
https://ipfs.io/

Installation

Install on macOS.

$ brew install ipfs

You might want to run a IPFS node in a Docker container.

# docker-compose.yml
version: "3"
services:
    # https://hub.docker.com/r/ipfs/go-ipfs/
    ipfs:
        image: ipfs/go-ipfs:v0.4.14
        working_dir: /export
        ports:
            - "4001:4001"
            - "5001:5001"
            - "8080:8080"
        volumes:
          - ./ipfs/data/ipfs:/data/ipfs
          - ./ipfs/export:/export

Start your IPFS node.

$ ipfs init
initializing IPFS node at /Users/vinta/.ipfs
generating 2048-bit RSA keypair... done
peer identity: QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6

$ ipfs daemon

ref:
https://ipfs.io/docs/commands/#ipfs-init
https://ipfs.io/docs/commands/#ipfs-daemon

Usage

Show Node Info

$ ipfs id
{
    "ID": "QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6",
    "PublicKey": "A_LONG_LONG_LONG_KEY,
    "Addresses": [
        "/ip4/127.0.0.1/tcp/4001/ipfs/QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6",
        "/ip4/172.19.0.2/tcp/4001/ipfs/QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6"
    ],
    "AgentVersion": "go-ipfs/0.4.14/5db3846",
    "ProtocolVersion": "ipfs/0.1.0"
}

ref:
https://ipfs.io/docs/getting-started/

Add Other Nodes to Your Bootstrap List

This one is from Muzeum, https://muzeum.pro/.

$ ipfs bootstrap add /ip4/52.221.121.238/tcp/4001/ipfs/QmTKYdZDkqHiY24kPynSmKbmRdk7cJxWsvvfvvvZArQ1N9

# you could also connect to a node directly
$ ipfs swarm connect /ip4/52.221.121.238/tcp/4001/ipfs/QmTKYdZDkqHiY24kPynSmKbmRdk7cJxWsvvfvvvZArQ1N9

ref:
https://ipfs.io/docs/commands/#ipfs-bootstrap
https://ipfs.io/docs/commands/#ipfs-swarm

Add Files to IPFS

Every IPFS node's default storage is 10GB, and a single node could only store data it needs, which also means each node only stores a small amount of whole data on IPFS. If there is not enough nodes, your data might be distributed to no one except your own node.

Your content is automatically pinned when you ipfs add it.

$ ipfs add -r mysite
added QmRticJ3P5fnb9GGnUj3U9XMkYvGEnv9AQfk6YmgRhivYA mysite/index.html
added QmY9cxiHqTFoWamkQVkpmmqzBrY3hCBEL2XNu3NtX74Fuu mysite/readme.md
added QmTLhFgeWLacpbiGNYmhchHGQAhfNyDZcLt5akJFFLV89V mysite

If files/folders under the folder change, the hash of the folder changes too.

$ vim mysite/index.html
$ ipfs add -r mysite
added QmQTTe3deLfeULKjPHnQTcyFuCmY5JZiwSTiPT4nSt1KVK mysite/index.html # changed
added QmS85tb3aKQNurFm51FaxtK6NyNei4ej3gDR21baDZXRoU mysite            # changed

ref:
https://ipfs.io/docs/commands/#ipfs-add

Pin Files from IPFS

Pinning means storing IPFS files on local node, and prevent them from getting garbage collected. Also, you could access them much quickly. You only need to do ipfs pin add to pin contents someone else uploaded.

$ ipfs pin add -r --progress /ipns/ipfs.soundscape.net/

$ ipfs pin add --progress /ipns/ipfs.soundscape.net/music_group/index.json
pinned QmZwTEhdjT4MyvEnWndVEJzBjp8zGGZH1cEBpshBQs75rY recursively

$ ipfs pin add --progress /ipns/ipfs.soundscape.net/music_album/index.json
pinned QmSAuGU5xt5SdR2ca2EDgeHFATSrAQhTfTYpYs9K9qmqED recursively

$ ipfs pin add --progress /ipns/ipfs.soundscape.net/music_recording/index.json
pinned QmcTiadA9jRMXx77tydPa6492QJAtjXkKkA4gERaFksy94 recursively

$ ipfs pin add --progress /ipns/ipfs.soundscape.net/music_composition/index.json
pinned QmTfqVaGVRnaPRQgYypGYXUvTK1UcDfK5VWYvU4rwK3m26 recursively

P.S. Sometimes when I ipfs pin add a file which is not on my node, the command just hangs there. I'm not sure why that once I access the file first (through curl or any browser), then ipfs pin add works fine. But it does not make sense: if I already get/access/download the file, I could just ipfs add the file and it would be automatically pinned.

ref:
https://ipfs.io/docs/commands/#ipfs-pin

Get Files

You have several ways to get files or folders from IPFS:

  • ipfs get dir-hash -o readable-dir-name
    • ipfs get QmbMQNcg8TTo5dXZPtuxbns1XVq6cZJaa7vNqZzeJpKwfk -o mysite
  • ipfs get file-hash -o readable-file-name.ext
    • ipfs get Qmd286K6pohQcTKYqnS1YhWrCiS4gz7Xi34sdwMe9USZ7u -o cat.jpg
  • ipfs get /ipfs/dir-hash/path/to/file.txt
    • ipfs get /ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPpHdWEz79ojWnPbdG/readme
  • ipfs get /ipns/example.com/path/to/file.txt
    • ipfs get /ipns/ipfs.soundscape.net/music_group/index.json

You could also access IPFS files through any public gateway:

  • curl https://ipfs.io/ipns/peer-id/path/to/file.txt
    • curl https://ipfs.io/ipns/QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6/index.html
  • curl https://ipfs.io/ipns/example.com/path/to/file.txt
    • curl https://ipfs.io/ipns/ipfs.soundscape.net/music_group/index.json
  • curl http://127.0.0.1:8080/ipns/example.com/path/to/file.txt
    • curl http://127.0.0.1:8080/ipns/ipfs.soundscape.net/music_group/index.json

Download IPFS objects with ipfs get.

$ ipfs ls QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ
Qmd286K6pohQcTKYqnS1YhWrCiS4gz7Xi34sdwMe9USZ7u 443362 cat.jpg

# you could get a folder
$ ipfs get QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ
$ ls QmW2WQi7j6c7UgJTarActp7tDNikE4B2qXtFCfLPdsgaTQ
cat.jpg

# as well as a file
$ ipfs get Qmd286K6pohQcTKYqnS1YhWrCiS4gz7Xi34sdwMe9USZ7u -o cat.jpg

# get files and rename them
$ mkdir -p soundscape/music_group/ soundscape/music_album/ soundscape/music_recording/ soundscape/music_composition/ && \
  ipfs get /ipns/ipfs.soundscape.net/music_group/index.json -o soundscape/music_group/index.json; \
  ipfs get /ipns/ipfs.soundscape.net/music_album/index.json -o soundscape/music_album/index.json; \
  ipfs get /ipns/ipfs.soundscape.net/music_recording/index.json -o soundscape/music_recording/index.json; \
  ipfs get /ipns/ipfs.soundscape.net/music_composition/index.json -o soundscape/music_composition/index.json

# get whole folders
$ ipfs get /ipns/ipfs.soundscape.net/music_group; \
  ipfs get /ipns/ipfs.soundscape.net/music_album; \
  ipfs get /ipns/ipfs.soundscape.net/music_recording; \
  ipfs get /ipns/ipfs.soundscape.net/music_composition

ref:
https://ipfs.io/docs/commands/#ipfs-get
https://discuss.ipfs.io/t/trying-to-better-understand-the-pinning-concept/754

Display IPFS object data with ipfs cat.

$ ipfs cat Qmd286K6pohQcTKYqnS1YhWrCiS4gz7Xi34sdwMe9USZ7u > cat.jpg
$ ipfs cat QmS4ustL54uo8FzR9455qaxZwuMiUhyvMcX9Ba8nUH4uVv/readme

Publish a Website to IPNS

IPNS stands for InterPlanetary Naming System.

Everytime you change files under a folder, the hash of the folder also changes. So you need a static reference which always points to the latest hash of your folder. You could publish your static website (a folder) to IPNS with the static reference, which is your peer ID as well as the hash of your public key.

By default, every IPFS node has only one pair of private and public key. Therefore, you could only publish one folder with your peer ID. But you could add new keypairs through ipfs key gen and publish multiple folders.

$ ipfs add -r mysite
added QmeqHWZgvgx5C7T6DakX75CJDRgAUoSDZayLYrcnAP8Fma mysite/index.html
added QmUtuRphD9rJgRkfxwj7DcyFEAcSeH3Q1fK8nHxxoDiKK5 mysite

$ ipfs name publish QmUtuRphD9rJgRkfxwj7DcyFEAcSeH3Q1fK8nHxxoDiKK5
published to QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6: /ipfs/QmUtuRphD9rJgRkfxwj7DcyFEAcSeH3Q1fK8nHxxoDiKK5

$ ipfs name resolve QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6
/ipfs/QmUtuRphD9rJgRkfxwj7DcyFEAcSeH3Q1fK8nHxxoDiKK5

Click following links to see contents.

After you change something, publish it again with new hash.

$ vim mysite/index.html
$ ipfs add -r mysite
added QmNjbhdks8RUgDt6QiNFe5QGe2HrbCsq5FKda9D9hLVkkU mysite/index.html # changed
added QmbMQNcg8TTo5dXZPtuxbns1XVq6cZJaa7vNqZzeJpKwfk mysite            # changed

$ ipfs name publish QmbMQNcg8TTo5dXZPtuxbns1XVq6cZJaa7vNqZzeJpKwfk
published to QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6: /ipfs/QmbMQNcg8TTo5dXZPtuxbns1XVq6cZJaa7vNqZzeJpKwfk

ref:
https://ipfs.io/docs/commands/#ipfs-name

Create a Domain Name Alias for Your Peer ID

The hash is not very friendly for humans. Fortunately, you could and probably should associate a domain name with your peer ID.

First, you need to add a TXT record whose value is dnslink=/ipns/YOUR_PEER_ID to your domain name. In the following article, we assume the domain name you choose is ipfs.kittenphile.com.

$ dig +short TXT ipfs.kittenphile.com
"dnslink=/ipns/QmfNy1th16zscbpxe8Q2EQdQkNFn7Y3Rp9kGZWL1EQDyw6"

$ ipfs name resolve -r ipfs.kittenphile.com
/ipfs/QmaE2DcNxGjPGPfzfTQuTBTW9D57abVSv319WqC89Av1y1

Click following links to see contents.

ref:
https://ipfs.io/docs/examples/example-viewer/example#../websites/README.md
https://hackernoon.com/ten-terrible-attempts-to-make-the-inter-planetary-file-system-human-friendly-e4e95df0c6fa

Public Gateway

If you have a public gateway and people retrieve files through it. Your public gateway fetches and stores the data, but it doesn't pin them. Files get removed with the next garbage collection run.

ref:
https://discuss.ipfs.io/t/public-facing-gateway-and-pinning/449

Deploy TICK stack on Kubernetes: Telegraf, InfluxDB, Chronograf, and Kapacitor

Deploy TICK stack on Kubernetes: Telegraf, InfluxDB, Chronograf, and Kapacitor

TICK stack is a set of open source tools for building a monitoring and alerting system, whose components include:

  • Telegraf: Collect data
  • InfluxDB: Store data
  • Chronograf: Visualize data
  • Kapacitor: Raise alerts

You could consider TICK as an alternative of ELK (Elasticsearch, Logstash, and Kibana).

ref:
https://www.influxdata.com/time-series-platform/

InfluxDB

InfluxDB is a Time Series Database which optimized for time-stamped or time series data. Time series data could be server metrics, application performance monitoring, network data, sensor data, events, clicks, trades in a market, and many other types of analytics data.

ref:
https://www.influxdata.com/time-series-platform/influxdb/

Key Concepts

  • Retention Policy: Database configurations which indicate how long a db keeps data and how many copies of those data are stored in the cluster.
  • Measurement: Conceptually similar to a RDBM table.
  • Point: Conceptually similar to a RDBM row.
  • Timestamp: Primary key of every point.
  • Field: Conceptually similar to a RDBM column (without indexes).
    • Field set menas a pair of Field key and Field value.
  • Tag: Basically indexed field.
  • Series: Data which is in the same measurement, retention policy, and tag set.

All time in InfluxDB is UTC.

ref:
https://docs.influxdata.com/influxdb/v1.5/concepts/key_concepts/
https://docs.influxdata.com/influxdb/v1.5/concepts/glossary/
https://docs.influxdata.com/influxdb/v1.5/concepts/crosswalk/
https://www.jianshu.com/p/a1344ca86e9b

Hardware Guideline

InfluxDB should be run on locally attached SSDs. Any other storage configuration will have lower performance characteristics and may not be able to recover from even small interruptions in normal processing.

ref:
https://docs.influxdata.com/influxdb/v1.5/guides/hardware_sizing/

Deployment

When you create a database, InfluxDB automatically creates a retention policy named autogen which has infinite retention. You may rename that retention policy or disable its auto-creation in the configuration file.

# tick/influxdb/service.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: tick
  name: influxdb
spec:
  clusterIP: None
  selector:
    app: influxdb
  ports:
  - name: api
    port: 8086
    targetPort: api
  - name: admin
    port: 8083
    targetPort: admin
# tick/influxdb/statefulset.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: tick
  name: influxdb
data:
  influxdb.conf: |+
    [meta]
      dir = "/var/lib/influxdb/meta"
      retention-autocreate = false
    [data]
      dir = "/var/lib/influxdb/data"
      engine = "tsm1"
      wal-dir = "/var/lib/influxdb/wal"
  init.iql: |+
    CREATE DATABASE "telegraf" WITH DURATION 90d REPLICATION 1 SHARD DURATION 1h NAME "rp_90d"
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: tick
  name: influxdb
spec:
  replicas: 1
  selector:
    matchLabels:
      app: influxdb
  serviceName: influxdb
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: ssd-ext4
      resources:
        requests:
          storage: 250Gi
  template:
    metadata:
      labels:
        app: influxdb
    spec:
      volumes:
      - name: config
        configMap:
          name: influxdb
          items:
            - key: influxdb.conf
              path: influxdb.conf
      - name: init-iql
        configMap:
          name: influxdb
          items:
            - key: init.iql
              path: init.iql
      containers:
      - name: influxdb
        image: influxdb:1.5.2-alpine
        ports:
        - name: api
          containerPort: 8086
        - name: admin
          containerPort: 8083
        volumeMounts:
        - name: data
          mountPath: /var/lib/influxdb
        - name: config
          mountPath: /etc/telegraf
        - name: init-iql
          mountPath: /docker-entrypoint-initdb.d
        resources:
          requests:
            cpu: 500m
            memory: 10G
          limits:
            cpu: 4000m
            memory: 10G
        readinessProbe:
          httpGet:
            path: /ping
            port: api
          initialDelaySeconds: 5
          timeoutSeconds: 5

ref:
https://hub.docker.com/_/influxdb/
https://docs.influxdata.com/influxdb/v1.5/administration/config/

$ kubectl apply -f tick/influxdb/ -R

Usage

$ kubectl get all --namespace tick
$ kubectl exec -i -t influxdb-0 --namespace tick -- influx
SHOW DATABASES
USE telegraf
SHOW MEASUREMENTS
SELECT * FROM diskio LIMIT 2
DROP MEASUREMENT access_log

ref:
https://docs.influxdata.com/influxdb/v1.5/query_language/data_download/
https://docs.influxdata.com/influxdb/v1.5/query_language/data_exploration/

Telegraf

Telegraf is a plugin-driven server agent process for collecting arbitrary metrics and writing them to multiple data storages include InfluxDB, Elasticsearch, CloudWatch, and so on.

ref:
https://www.influxdata.com/time-series-platform/telegraf/

Deployment

Collect system metrics per node using DaemonSet:

# tick/telegraf/daemonset.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: tick
  name: telegraf-ds
data:
  telegraf.conf: |+
    [agent]
      interval = "10s"
      round_interval = true
      metric_batch_size = 1000
      metric_buffer_limit = 10000
      collection_jitter = "0s"
      flush_interval = "10s"
      flush_jitter = "0s"
      precision = ""
      debug = true
      quiet = false
      logfile = ""
      hostname = "$HOSTNAME"
      omit_hostname = false
    [[outputs.influxdb]]
      urls = ["http://influxdb.tick.svc.cluster.local:8086"]
      database = "telegraf"
      retention_policy = "rp_90d"
      write_consistency = "any"
      timeout = "5s"
      username = ""
      password = ""
      user_agent = "telegraf"
      insecure_skip_verify = false
    [[inputs.cpu]]
      percpu = true
      totalcpu = true
      collect_cpu_time = false
    [[inputs.disk]]
      ignore_fs = ["tmpfs", "devtmpfs"]
    [[inputs.diskio]]
    [[inputs.docker]]
      endpoint = "unix:///var/run/docker.sock"
      container_names = []
      timeout = "5s"
      perdevice = true
      total = false
    [[inputs.kernel]]
    [[inputs.kubernetes]]
      url = "http://$HOSTNAME:10255"
      bearer_token = "/var/run/secrets/kubernetes.io/serviceaccount/token"
      insecure_skip_verify = true
    [[inputs.mem]]
    [[inputs.processes]]
    [[inputs.swap]]
    [[inputs.system]]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  namespace: tick
  name: telegraf-ds
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 3
  selector:
    matchLabels:
      app: telegraf
      type: ds
  template:
    metadata:
      labels:
        app: telegraf
        type: ds
    spec:
      containers:
      - name: telegraf
        image: telegraf:1.5.3-alpine
        env:
        - name: HOSTNAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: "HOST_PROC"
          value: "/rootfs/proc"
        - name: "HOST_SYS"
          value: "/rootfs/sys"
        volumeMounts:
        - name: sys
          mountPath: /rootfs/sys
          readOnly: true
        - name: proc
          mountPath: /rootfs/proc
          readOnly: true
        - name: docker-socket
          mountPath: /var/run/docker.sock
        - name: varrunutmp
          mountPath: /var/run/utmp
          readOnly: true
        - name: varlog
          mountPath: /var/log
          readOnly: true
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config
          mountPath: /etc/telegraf
          readOnly: true
        resources:
          requests:
            cpu: 50m
            memory: 500Mi
          limits:
            cpu: 200m
            memory: 500Mi
      volumes:
      - name: sys
        hostPath:
          path: /sys
      - name: docker-socket
        hostPath:
          path: /var/run/docker.sock
      - name: proc
        hostPath:
          path: /proc
      - name: varrunutmp
        hostPath:
          path: /var/run/utmp
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config
        configMap:
          name: telegraf-ds

ref:
https://github.com/influxdata/telegraf/blob/master/docs/CONFIGURATION.md
https://github.com/influxdata/telegraf/blob/master/plugins/inputs/docker/README.md
https://github.com/influxdata/telegraf/blob/master/plugins/inputs/kubernetes/README.md

Collect arbitrary metrics:

# tick/telegraf/deployment.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: tick
  name: telegraf-infra
data:
  telegraf.conf: |+
    [agent]
      interval = "10s"
      round_interval = true
      metric_batch_size = 1000
      metric_buffer_limit = 10000
      collection_jitter = "0s"
      flush_interval = "10s"
      flush_jitter = "0s"
      precision = ""
      debug = true
      quiet = false
      logfile = ""
      hostname = "telegraf-infra"
      omit_hostname = false
    [[outputs.influxdb]]
      urls = ["http://influxdb.tick.svc.cluster.local:8086"]
      database = "telegraf"
      retention_policy = "rp_90d"
      write_consistency = "any"
      timeout = "5s"
      username = ""
      password = ""
      user_agent = "telegraf"
      insecure_skip_verify = false
    [[inputs.http_listener]]
      service_address = ":8186"
    [[inputs.socket_listener]]
      service_address = "udp://:8092"
      data_format = "influx"
    [[inputs.redis]]
      servers = ["tcp://redis-cache.default.svc.cluster.local", "tcp://redis-broker.default.svc.cluster.local"]
    [[inputs.mongodb]]
      servers = ["mongodb://mongodb.default.svc.cluster.local"]
      gather_perdb_stats = true
---
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: tick
  name: telegraf-infra
spec:
  replicas: 1
  selector:
    matchLabels:
      app: telegraf
      type: infra
  template:
    metadata:
      labels:
        app: telegraf
        type: infra
    spec:
      containers:
      - name: telegraf
        image: telegraf:1.5.3-alpine
        ports:
        - name: udp
          protocol: UDP
          containerPort: 8092
        - name: http
          containerPort: 8186
        volumeMounts:
        - name: config
          mountPath: /etc/telegraf
        resources:
          requests:
            cpu: 50m
            memory: 500Mi
          limits:
            cpu: 500m
            memory: 500Mi
      volumes:
      - name: config
        configMap:
          name: telegraf-infra

ref:
https://github.com/influxdata/telegraf/blob/master/plugins/inputs/exec/README.md
https://github.com/influxdata/telegraf/blob/master/plugins/inputs/mongodb/README.md
https://github.com/influxdata/telegraf/blob/master/plugins/inputs/redis/README.md

$ kubectl apply -f tick/telegraf/ -R

Furthermore, you might also want to parse stdout of all containers, in that case, you could use logparser with DaemonSet:

# telegraf.conf
[agent]
    interval = "1s"
    round_interval = true
    metric_batch_size = 1000
    metric_buffer_limit = 10000
    collection_jitter = "0s"
    flush_interval = "10s"
    flush_jitter = "0s"
    precision = ""
    debug = true
    quiet = false
    logfile = ""
    hostname = "$HOSTNAME"
    omit_hostname = false
[[outputs.file]]
  files = ["stdout"]
[[outputs.influxdb]]
    urls = ["http://influxdb.tick.svc.cluster.local:8086"]
    database = "your_db"
    retention_policy = "rp_90d"
    write_consistency = "any"
    timeout = "5s"
    username = ""
    password = ""
    user_agent = "telegraf"
    insecure_skip_verify = false
    namepass = ["logparser_*"]
[[inputs.logparser]]
    name_override = "logparser_api"
    files = ["/var/log/containers/api*.log"]
    from_beginning = false
    [inputs.logparser.grok]
    measurement = "api_access_log"
    patterns = ["bytes\\} \\[%{DATA:timestamp:ts-ansic}\\] %{WORD:request_method} %{URIPATH:request_path}%{DATA:request_params:drop} =\\\\u003e generated %{NUMBER:response_bytes:int} bytes in %{NUMBER:response_time_ms:int} msecs \\(HTTP/1.1 %{RESPONSE_CODE}"]
[[inputs.logparser]]
    name_override = "logparser_worker"
    files = ["/var/log/containers/worker*.log"]
    from_beginning = false
    [inputs.logparser.grok]
    measurement = "worker_task_log"
    patterns = ['''\[%{TIMESTAMP_ISO8601:timestamp:ts-"2006-01-02 15:04:05"},%{WORD:value1:drop}: %{LOGLEVEL:loglevel:tag}\/MainProcess\] Task %{PROG:task_name:tag}\[%{UUID:task_id:drop}\] %{WORD:execution_status:tag} in %{DURATION:execution_time:duration}''']

ref:
https://github.com/influxdata/telegraf/tree/master/plugins/inputs/logparser
https://github.com/influxdata/telegraf/blob/master/plugins/inputs/logparser/grok/patterns/influx-patterns
https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

Debug telegraf.conf

$ docker run \
-v $PWD/telegraf.conf:/etc/telegraf/telegraf.conf \
-v $PWD/api-1.log:/var/log/containers/api-1.log \
-v $PWD/worker-1.log:/var/log/containers/worker-1.log \
telegraf

ref:
https://grokdebug.herokuapp.com/

Usage

from telegraf.client import TelegrafClient

client = TelegrafClient(host='telegraf.tick.svc.cluster.local', port=8092)
client.metric('some_measurement', {'value_a': 100, 'value_b': 0, 'value_c': True}, tags={'country': 'taiwan'})

ref:
https://github.com/paksu/pytelegraf

Kapacitor

Kapacitor is so-called a real-time streaming data processing engine, basically, you would use it to trigger alerts.

ref:
https://www.influxdata.com/time-series-platform/kapacitor/

Deployment

# tick/kapacitor/service.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: tick
  name: kapacitor-ss
spec:
  clusterIP: None
  selector:
    app: kapacitor
  ports:
  - name: api
    port: 9092
    targetPort: api
---
apiVersion: v1
kind: Service
metadata:
  namespace: tick
  name: kapacitor
spec:
  type: ClusterIP
  selector:
    app: kapacitor
  ports:
  - name: api
    port: 9092
    targetPort: api
# tick/kapacitor/statefulset.yaml


apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: tick
  name: kapacitor
spec:
  replicas: 1
  serviceName: kapacitor-ss
  selector:
    matchLabels:
      app: kapacitor
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: hdd-ext4
      resources:
        requests:
          storage: 10Gi
  template:
    metadata:
      labels:
        app: kapacitor
    spec:
      containers:
      - name: kapacitor
        image: kapacitor:1.4.1-alpine
        env:
        - name: KAPACITOR_HOSTNAME
          value: kapacitor
        - name: KAPACITOR_INFLUXDB_0_URLS_0
          value: http://influxdb.tick.svc.cluster.local:8086
        ports:
        - name: api
          containerPort: 9092
        volumeMounts:
        - name: data
          mountPath: /var/lib/kapacitor
        resources:
          requests:
            cpu: 50m
            memory: 500Mi
          limits:
            cpu: 500m
            memory: 500Mi

ref:
https://docs.influxdata.com/kapacitor/v1.4/

$ kubectl apply -f tick/kapacitor/ -R

Chronograf

Chronograf is the Web UI for TICK stack.

ref:
https://www.influxdata.com/time-series-platform/chronograf/

Deployment

# tick/chronograf/service.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: tick
  name: chronograf-ss
spec:
  clusterIP: None
  selector:
    app: chronograf
---
apiVersion: v1
kind: Service
metadata:
  namespace: tick
  name: chronograf
spec:
  selector:
    app: chronograf
  ports:
  - name: api
    port: 80
    targetPort: api
# tick/chronograf/statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  namespace: tick
  name: chronograf
spec:
  replicas: 1
  serviceName: chronograf-ss
  selector:
    matchLabels:
      app: chronograf
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: ["ReadWriteOnce"]
      storageClassName: hdd-ext4
      resources:
        requests:
          storage: 10Gi
  template:
    metadata:
      labels:
        app: chronograf
    spec:
      containers:
      - name: chronograf
        image: chronograf:1.4.4.0-alpine
        command: ["chronograf"]
        args: ["--influxdb-url=http://influxdb.tick.svc.cluster.local:8086", "--kapacitor-url=http://kapacitor.tick.svc.cluster.local:9092"]
        ports:
        - name: api
          containerPort: 8888
        livenessProbe:
          httpGet:
            path: /ping
            port: api
        readinessProbe:
          httpGet:
            path: /ping
            port: api
        volumeMounts:
        - name: data
          mountPath: /var/lib/chronograf
        resources:
          requests:
            cpu: 100m
            memory: 1000Mi
          limits:
            cpu: 2000m
            memory: 1000Mi

ref:
https://docs.influxdata.com/chronograf/v1.4/

$ kubectl apply -f tick/chronograf/ -R
$ kubectl port-forward svc/chronograf 8888:80 --namespace tick

ref:
https://kubernetes.io/docs/tasks/access-application-cluster/port-forward-access-application-cluster/

Pipenv and Pipfile: The officially recommended Python packaging tool

Pipenv and Pipfile: The officially recommended Python packaging tool

You no longer need to use pip and virtualenv separately. Use pipenv instead.

ref:
https://docs.pipenv.org/

Install

$ pip install pipenv

ref:
https://github.com/pypa/pipenv

Usage

Caution: Pipenv is not compatible with Anaconda.

$ pyenv global 3.6.4

# initialize project virtualenv with a specific Python version
# automatically generate both Pipfile and Pipfile.lock from an existed requirements.txt
$ pipenv install --python python3

$ cd /path/to/project-contains-Pipfile
$ pipenv install

$ pipenv install pangu
$ pipenv install -r requirements.txt

# install packages to dev-packages
$ pipenv install --dev \
ipdb \
flake8 \
flake8-bandit \
flake8-bugbear \
flake8-comprehensions \
flake8-debugger \
flake8-print \
flake8-string-format \
pep8-naming \
pylint \
pylint-common \
pylint-celery \
pylint-django \
pylint-flask

# switch your shell environment to project virtualenv
$ pipenv shell
$ exit

# uninstall everything
$ pipenv uninstall --all

# remove project virtualenv
$ pipenv --rm

ref:
https://docs.pipenv.org/basics/
https://docs.pipenv.org/advanced/

Example Pipfile

[[source]]
url = "https://pypi.python.org/simple" 
verify_ssl = true 
name = "pypi" 

[packages] 
requests = ">=2.0.0" 

[dev-packages] 
flake8 = "*" 
ipdb = "*" 
ipython = "*" 
pylint = "*" 

[requires] 
python_version = "3.6" 
Print traceback call stack in Python

Print traceback call stack in Python

Print Python stack traces without raising an exception.

# in /path/to/a/module.py
import traceback
for line in traceback.format_stack():
    print(line.strip())

# or

print('----- start -----')
import traceback; traceback.print_stack()
print('----- end -----')
>>> import qingcloud.iaas
>>> conn = qingcloud.iaas.connect_to_zone('pek2', '123', '456')
>>> conn.describe_instances(limit=1)
----- start -----
  File "<stdin>", line 1, in <module>
  File "qingcloud/iaas/connection.py", line 214, in describe_instances
    return self.send_request(action, body)
  File "qingcloud/iaas/connection.py", line 42, in send_request
    resp = self.send(url, request, verb)
  File "qingcloud/conn/connection.py", line 245, in send
    request.authorize(self)
  File "qingcloud/conn/connection.py", line 156, in authorize
    connection._auth_handler.add_auth(self, **kwargs)
  File "qingcloud/conn/auth.py", line 118, in add_auth
    import traceback; traceback.print_stack()
----- end -----

ref:
https://stackoverflow.com/questions/3925248/print-python-stack-trace-without-exception-being-raised