Service Discovery with Consul and consul-template

Consul agent 必須裝在 cluster 裡的每一台機器上
agent 可以是以 server mode 或 client mode 運行
官方的建議是每個 datacenter 裡至少有 3-5 個 agent server

agent server 負責串連整個 clusters 和紀錄 nodes 資訊
agent client 負責 register service, run health check, forward queris to servers

ref:
https://consul.io/intro/getting-started/agent.html

Installation

$ sudo apt-get install unzip
$ wget https://dl.bintray.com/mitchellh/consul/0.5.0_linux_amd64.zip
$ unzip 0.5.0_linux_amd64.zip
$ sudo mv consul /usr/local/bin/

ref:
https://www.consul.io/intro/getting-started/install.html

Usage

You can get your Atlas token from https://atlas.hashicorp.com/settings/tokens.

# the first node
$ consul agent -server -bootstrap-expect 2 -data-dir /tmp/consul -config-dir /etc/consul.d \
-atlas=vinta/consul-test \
-atlas-token="YOUR_TOKEN" \
-atlas-join

# other nodes: server mode
$ consul agent -server -data-dir /tmp/consul -config-dir /etc/consul.d \
-atlas=vinta/consul-test \
-atlas-token="YOUR_TOKEN" \
-atlas-join

# other nodes: client model
$ consul agent -data-dir /tmp/consul -config-dir /etc/consul.d \
-atlas=vinta/consul-test \
-atlas-token="YOUR_TOKEN" \
-atlas-join

# if we want to access the web UI remotely
# we'll have to specify the public IP of the client (104.236.148.236)
$ consul agent -data-dir /tmp/consul -config-dir /etc/consul.d \
-client 104.236.148.236 \
-ui-dir /root/consul_web_ui/dist \

# when you don't specify `-join`
# run `consul join NEW_NODE_IP` command on any node (servers or clients) of this cluster
$ consul join 10.134.249.144

$ consul members

# you can query services on any node of this cluster
$ http http://localhost:8500/v1/catalog/nodes
$ http http://localhost:8500/v1/catalog/services
$ http http://localhost:8500/v1/catalog/service/salt-master
$ http http://localhost:8500/v1/catalog/service/nginx
$ http http://localhost:8500/v1/health/service/nginx?passing

ref:
https://consul.io/docs/agent/options.html
https://consul.io/docs/guides/atlas.html
https://consul.io/docs/agent/http.html

Service Definition

in /etc/consul.d/nginx.json

server 和 client 都可以有 service

address 欄位默認就是執行該 consul agent 的機器的 IP,所以可以不填

{
  "service": {
    "name": "nginx",
    "port": 80,
    "check": {
      "script": "curl localhost > /dev/null 2>&1",
      "interval": "5s"
    }
  }
}
$ consul reload

$ http http://localhost:8500/v1/catalog/service/nginx

ref:
https://consul.io/docs/agent/services.html
https://consul.io/docs/agent/checks.html

Integration with consul-template

Take HAProxy as an example.

$ wget https://github.com/hashicorp/consul-template/releases/download/v0.8.0/consul-template_0.8.0_linux_amd64.tar.gz
$ tar -zxvf consul-template_0.8.0_linux_amd64.tar.gz
$ sudo mv consul-template_0.8.0_linux_amd64/consul-template /usr/local/bin/

in /etc/consul-template/haproxy.conf

consul = "127.0.0.1:8500"

syslog {
  enabled = true
}

template {
  source = "/etc/consul-template/haproxy.cfg.ctmpl"
  destination = "/etc/haproxy/haproxy.cfg"
  command = "sudo service haproxy reload"
}

in /etc/consul-template/haproxy.cfg.ctmpl

...
backend web_nodes
    mode http
    balance roundrobin
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    cookie haproxyserverid insert nocache maxidle 1h
    option httpchk HEAD /
    {{ range service "nginx" }}
    server {{.Node}} {{.Address}}:{{.Port}} check cookie {{.Node}}
    {{end}}
...
# -dry: Dump generated templates to stdout
$ consul-template -config /etc/consul-template/haproxy.conf -dry

ref:
https://github.com/hashicorp/consul-template

Integration with confd

confd 只支援 consul 的 Key/Value store API
不支援 service, health check 的 API
建議用 consul-template
https://github.com/kelseyhightower/confd/pull/102

ref:
https://github.com/kelseyhightower/confd