Archive for April 2015

Install Go (Golang)

Installation

Ubuntu

$ sudo apt-get install -V golang
# or
$ wget https://storage.googleapis.com/golang/go1.4.2.linux-amd64.tar.gz
$ tar -C /usr/local -xzf go1.4.2.linux-amd64.tar.gz
# then add `export PATH=$PATH:/usr/local/go/bin` to your `/etc/profile`

$ which go

Mac OS X

$ brew install mercurial
$ bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)
# then add `source /Users/vinta/.gvm/scripts/gvm` to your `.bash_prfile` or `.zshrc`

$ gvm listall
$ gvm install go1.4.2
$ gvm list
$ gvm use go1.4.2 --default

$ go env

ref:
https://github.com/moovweb/gvm

Environment Variables

GOROOT
就是你的 go 的安裝目錄(即是 bin/go 所在的路徑)
例如 /Users/vinta/.gvm/gos/go1.4.2

GOPATH
就是你的 workspace
例如 /Users/vinta/.gvm/pkgsets/go1.4.2/global
這個路徑你可以隨意指定()
go get 會把原始碼放到 $GOPATH/src
執行檔放到 $GOPATH/bin
lib (.a) 放到 $GOPATH/pkg

ref:
http://golang.org/doc/code.html

Read and save file in Django / Python

File 和 ImageFile 接受 Python 的 file 或 StringIO 物件
而 ContentFile 接受 string

ref:
https://docs.djangoproject.com/en/dev/ref/files/file/#the-file-object

Django Form

image_file = request.FILES['file']

# 方法一
profile.mugshot.save(image_file.name, image_file)

# 方法二
profile.mugshot = image_file

profile.save()

ref:
File Upload with Form in Django

open('/path/to/file.png')

from django.core.files import File

with open('/home/vinta/image.png', 'rb') as f:
    profile.mugshot = File(f)
    profile.save()

Django ContentFile

import os
import uuid

from django.core.files.base import ContentFile

import requests

url = 'http://vinta.ws/static/photo.jpg'
r = requests.get(url)
file_url, file_ext = os.path.splitext(r.url)
file_name = '%s%s' % (str(uuid.uuid4()).replace('-', ''), file_ext)

profile.mugshot.save('123.png', ContentFile(r.content), save=False)

# 如果 profile.mugshot 是 ImageField 欄位的話
# 可以用以下的方式來判斷它是不是合法的圖檔
try:
    profile.mugshot.width
except TypeError:
    raise RuntimeError('圖檔格式不正確')

profile.save()

Data URI, Base64

from binascii import a2b_base64

from django.core.files.base import ContentFile

data_uri = 'data:image/jpeg;base64,/9j/4AAQSkZJRg....'
head, data = data_uri.split(',')
binary_data = a2b_base64(data)

# 方法一
profile.mugshot.save('whatever.jpg', ContentFile(binary_data), save=False)
profile.save()

# 不能用這種方式,因為少了 file name
profile.mugshot = ContentFile(binary_data)
profile.save()

# 方法二
f = open('image.png', 'wb')
f.write(binary_data)
f.close()

# 方法三
from StringIO import StringIO
from PIL import Image
img = Image.open(StringIO(binary_data))
print img.size

ref:
http://stackoverflow.com/questions/19395649/python-pil-create-and-save-image-from-data-uri

StringIO, PIL image

你就把 StringIO 想成是 open('/home/vinta/some_file.txt', 'rb') 的 file 物件

from StringIO import StringIO

from PIL import Image
import requests

r = requests.get('http://vinta.ws/static/photo.jpg')
img = Image.open(StringIO(r.content))
print pil_image.size

StringIO, PIL image, Django

from StringIO import StringIO

from django.core.files.base import ContentFile

from PIL import Image

raw_img_io = StringIO(binary_data)
img = Image.open(raw_img_io)
img = img.resize((524, 328), Image.ANTIALIAS)
img_io = StringIO()
img.save(img_io, 'PNG', quality=100)

profile.image.save('whatever.png', ContentFile(img_io.getvalue()), save=False)
profile.save()

ref:
http://stackoverflow.com/questions/3723220/how-do-you-convert-a-pil-image-to-a-django-file

Download file from URL, tempfile

import os
import tempfile
import requests
import xlrd

try:
    file_path = report.file.path
    temp = None
except NotImplementedError:
    url = report.file.url
    r = requests.get(url, stream=True)
    file_url, file_ext = os.path.splitext(r.url)

    # delete=True 會在 temp.close() 之後自己刪掉
    temp = tempfile.NamedTemporaryFile(prefix='report_file_', suffix=file_ext, dir='/tmp', delete=False)
    file_path = temp.name

    with open(file_path, 'wb') as f:
        for chunk in r.iter_content(chunk_size=1024):
            if chunk:
                f.write(chunk)
                f.flush()

wb = xlrd.open_workbook(file_path)

...

# 因為是 tempfile.NamedTemporaryFile(delete=False)
# 所以你要自己刪掉
try:
    os.remove(temp.name)
except AttributeError:
    pass

ref:
http://stackoverflow.com/questions/16694907/how-to-download-large-file-in-python-with-requests-py
http://pymotw.com/2/tempfile/

Service Discovery with Consul and consul-template

Consul agent 必須裝在 cluster 裡的每一台機器上
agent 可以是以 server mode 或 client mode 運行
官方的建議是每個 datacenter 裡至少有 3-5 個 agent server

agent server 負責串連整個 clusters 和紀錄 nodes 資訊
agent client 負責 register service, run health check, forward queris to servers

ref:
https://consul.io/intro/getting-started/agent.html

Installation

$ sudo apt-get install unzip
$ wget https://dl.bintray.com/mitchellh/consul/0.5.0_linux_amd64.zip
$ unzip 0.5.0_linux_amd64.zip
$ sudo mv consul /usr/local/bin/

ref:
https://www.consul.io/intro/getting-started/install.html

Usage

You can get your Atlas token from https://atlas.hashicorp.com/settings/tokens.

# the first node
$ consul agent -server -bootstrap-expect 2 -data-dir /tmp/consul -config-dir /etc/consul.d \
-atlas=vinta/consul-test \
-atlas-token="YOUR_TOKEN" \
-atlas-join

# other nodes: server mode
$ consul agent -server -data-dir /tmp/consul -config-dir /etc/consul.d \
-atlas=vinta/consul-test \
-atlas-token="YOUR_TOKEN" \
-atlas-join

# other nodes: client model
$ consul agent -data-dir /tmp/consul -config-dir /etc/consul.d \
-atlas=vinta/consul-test \
-atlas-token="YOUR_TOKEN" \
-atlas-join

# if we want to access the web UI remotely
# we'll have to specify the public IP of the client (104.236.148.236)
$ consul agent -data-dir /tmp/consul -config-dir /etc/consul.d \
-client 104.236.148.236 \
-ui-dir /root/consul_web_ui/dist \

# when you don't specify `-join`
# run `consul join NEW_NODE_IP` command on any node (servers or clients) of this cluster
$ consul join 10.134.249.144

$ consul members

# you can query services on any node of this cluster
$ http http://localhost:8500/v1/catalog/nodes
$ http http://localhost:8500/v1/catalog/services
$ http http://localhost:8500/v1/catalog/service/salt-master
$ http http://localhost:8500/v1/catalog/service/nginx
$ http http://localhost:8500/v1/health/service/nginx?passing

ref:
https://consul.io/docs/agent/options.html
https://consul.io/docs/guides/atlas.html
https://consul.io/docs/agent/http.html

Service Definition

in /etc/consul.d/nginx.json

server 和 client 都可以有 service

address 欄位默認就是執行該 consul agent 的機器的 IP,所以可以不填

{
  "service": {
    "name": "nginx",
    "port": 80,
    "check": {
      "script": "curl localhost > /dev/null 2>&1",
      "interval": "5s"
    }
  }
}
$ consul reload

$ http http://localhost:8500/v1/catalog/service/nginx

ref:
https://consul.io/docs/agent/services.html
https://consul.io/docs/agent/checks.html

Integration with consul-template

Take HAProxy as an example.

$ wget https://github.com/hashicorp/consul-template/releases/download/v0.8.0/consul-template_0.8.0_linux_amd64.tar.gz
$ tar -zxvf consul-template_0.8.0_linux_amd64.tar.gz
$ sudo mv consul-template_0.8.0_linux_amd64/consul-template /usr/local/bin/

in /etc/consul-template/haproxy.conf

consul = "127.0.0.1:8500"

syslog {
  enabled = true
}

template {
  source = "/etc/consul-template/haproxy.cfg.ctmpl"
  destination = "/etc/haproxy/haproxy.cfg"
  command = "sudo service haproxy reload"
}

in /etc/consul-template/haproxy.cfg.ctmpl

...
backend web_nodes
    mode http
    balance roundrobin
    option forwardfor
    http-request set-header X-Forwarded-Port %[dst_port]
    http-request add-header X-Forwarded-Proto https if { ssl_fc }
    cookie haproxyserverid insert nocache maxidle 1h
    option httpchk HEAD /
    {{ range service "nginx" }}
    server {{.Node}} {{.Address}}:{{.Port}} check cookie {{.Node}}
    {{end}}
...
# -dry: Dump generated templates to stdout
$ consul-template -config /etc/consul-template/haproxy.conf -dry

ref:
https://github.com/hashicorp/consul-template

Integration with confd

confd 只支援 consul 的 Key/Value store API
不支援 service, health check 的 API
建議用 consul-template
https://github.com/kelseyhightower/confd/pull/102

ref:
https://github.com/kelseyhightower/confd