碼天狗週刊 第 140 期 @vinta - MongoDB, Kubernetes, NGINX, Google Cloud Platform, MySQL

碼天狗週刊 第 140 期 @vinta - MongoDB, Kubernetes, NGINX, Google Cloud Platform, MySQL

本文同步發表於 CodeTengu Weekly - Issue 140

MongoDB cookbook: Queries and Aggregations

Issue 130 有提到,MongoDB 的 Aggregation 其實很強大,尤其搭配 $elemMatch$project$let$unwind$facet 等功能,可以直接完成很多複雜的業務邏輯,不需要多寫一行 code,雖然哪些事應該讓 DB 做、哪些事得在 API server 做,這就見仁見智啦。

不過 MongoDB Aggregation 寫起來的阿雜程度實在也跟 Elasticsearch 的 Query DSL 不遑多讓了(Thanks JSON),因為老是記不起來各種 operators 的用法和限制,所以就遵循之前提過的 Cookbook 模式,幫自己寫了一份筆記,複習、速查、分享各相宜。

Kubernetes Best Practices with Sandeep Dinesh (Google)

這個影片是 Google 的工程師在講使用 Kubernetes 和 containers 時的最佳實踐,影片的後半段則是 Weaveworks 的人在講他們搭建自己的 Kubernetes cluster 時遇到的各種挑戰和解法。

雖然前半段的內容有不少在 Kubernetes 和 GKE 的官方文件裡都有提到,但是有人貼心地幫你整理好還是挺棒的(就像你訂閱的這個 weekly 一樣),畢竟 Kubernetes 的文件真心多到靠北,看完都已經是 YAML 的形狀了。不過我對於越來越多人都推薦 Helm 這點還是不太能領略,總覺得 Helm 對一般使用者的意義好像不大啊(又不是 PaaS),我還不如直接幹一份 Chart 回來自己維護,之後要升級或客製化也比較方便,畢竟也就是一堆 YAML 檔。比較可行的用途似乎是團隊共用一套 Chart 來部署 production、staging 或 dev 環境?

延伸閱讀:

Tuning NGINX behind Google Cloud Platform HTTP(S) Load Balancer

因為 Google Cloud HTTP Load Balancing 的某些特性,如果你在 Google Kubernetes Engine 裡面跑 NGINX(或 OpenResty)的話,會有一些額外的 config 需要設定,尤其是 keepalive_timeout 620s;

題外話,Google Cloud 的 Load Balancer 也是很強啊,除了支援 QUIC 之外,更是默認啟用 TCP BBR

延伸閱讀:

别废话,各种 SQL 到底加了什么锁?

這個系列的文章專門在講 MySQL InnoDB 在各種情況下會使用的各種 lock,作者寫得非常淺顯易懂,最喜歡讀這種技術文章了~

延伸閱讀:

TeePublic

上禮拜發現的一個專門賣 T-shirt 的網站,重點是上面賣的 T-shirt 都!超!宅!它甚至有一個叫做 Programmer 的分類,或是你也可以隨便拿幾個你喜歡的電影、遊戲或動漫畫作品的名字去搜尋看看,保證有驚喜。我看到的第一天就買了八件。推薦各位臭宅去感受一下。

@vinta 分享!

Force HTTPS when using AWS ELB and nginx

You only have to setup the SSL certificate on Amazon ELB instead of every EC2 instance that behind ELB, normally you only receive HTTP traffic from Amazon ELB that serves both HTTP and HTTPS endpoints. The problem is that forcing HTTPS will result in a redirect loop when nginx only listens 80 port. A solution is forwarding HTTPS requests to another port, like 1443, in Amazon ELB.

Amazon ELB

Listeners configurations:

HTTP 80 >> HTTP 80
HTTPS 443 >> HTTP 1443

and health check target port should be HTTP:1443 instead of HTTP:80 or HTTPS:443.

nginx

in /etc/nginx/sites-available/your_website.conf

server {
    listen 80;
    server_name yahoo.streetvoice.com;
    rewrite ^ https://$host$request_uri? permanent;
}

server {
    listen 1443;
    server_name yahoo.streetvoice.com;

    ...
}

ref:
http://serverfault.com/questions/619971/redirect-all-http-requests-behind-amazon-elb-to-https-without-using-if
http://scottwb.com/blog/2013/10/28/always-on-https-with-nginx-behind-an-elb/

nginx as load balancer

WSGI using uWSGI and nginx on Ubuntu
https://library.linode.com/web-servers/nginx/python-uwsgi/ubuntu-12.04-precise-pangolin

HTTP 负载均衡模块(HTTP Upstream)
http://www.howtocn.org/nginx:nginx%E6%A8%A1%E5%9D%97%E5%8F%82%E8%80%83%E6%89%8B%E5%86%8C%E4%B8%AD%E6%96%87%E7%89%88:standardhttpmodules:httpupstream

nginx 負載平衡的策略
http://wenku.baidu.com/view/175894c708a1284ac850438a.html

Nginx 模块推荐 Session 粘连
http://www.php-oa.com/2012/03/15/nginx-sticky-upstream-check.html

使用 nginx sticky 实现基于 cookie 的负载均衡
http://www.ttlsa.com/nginx/nginx-modules-nginx-sticky-module/

ref:
http://blog.csdn.net/ydt619/article/details/5954632
http://www.wubin.org.cn/?action=show&id=78

svcn-web1 自己當 nginx load balancer

upstream django_cluster {
    ip_hash;
    # or
    least_conn;
    server 100.100.100.70:8000 weight=3; # svtw-web1
    server 100.100.100.71:8000 weight=4; # svtw-web2
    server 100.100.100.72:8000 weight=4; # svtw-web3
    ...
}

server {
    listen  80;
    server_name  streetvoice.cn;
    charset  utf-8;
    client_max_body_size  75M;

    location /asset  {
        alias /data/storage/asset;
        access_log off;
    }

    location / {
        real_ip_header      X-Forwarded-For;
        set_real_ip_from    10.0.0.0/8;
        proxy_set_header    Host $http_host;
        proxy_redirect      off;
        proxy_read_timeout  120;
        include             /etc/nginx/uwsgi_params;
        uwsgi_pass          django_cluster;
    }
}

weight 默認為 1
max_fails 默認為 1
fail_timeout 默認為 10s

基本上默認的配置就夠了

意思是那台機器發生 max_fails 次錯誤的話
在 fail_timeout 內會被標成不可用

ref:
http://nginx.org/en/docs/http/ngx_http_upstream_module.html#server

Compile nginx with sticky-module on Ubuntu 12.04

# 先移除用 apt-get 安裝的 nginx
$ sudo apt-get autoremove nginx

開始編譯

$ sudo su root

$ cd /usr/src/
$ apt-get source nginx

$ cd nginx-1.6.0/debian/modules
$ wget https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng/get/1.2.5.tar.gz
$ tar zxvf 1.2.5.tar.gz
$ mv nginx-goodies-nginx-sticky-module-ng-bd312d586752/ nginx-goodies-nginx-sticky-module-ng/

$ vim /usr/src/nginx-1.6.0/debian/rules
# 在 full_configure_flags 底下加上:
# --add-module=$(MODULESDIR)/ngx_http_substitutions_filter_module \
# --add-module=$(MODULESDIR)/nginx-goodies-nginx-sticky-module-ng

$ cd /usr/src/nginx-1.6.0/
$ aptitude build-dep nginx
$ aptitude install liblua5.1-0-dev init-system-helpers
$ dpkg-buildpackage -b

$ cd /usr/src
$ dpkg -i nginx-common_1.6.0-1+precise0_all.deb
$ dpkg -i nginx-full_1.6.0-1+precise0_amd64.deb

# 驗證一下
$ nginx -V

ref:
http://gravitronic.com/compiling-the-nginx-sticky-session-module-in-ubuntu/

Install WordPress with nginx on Ubuntu 14.04 (LEMP)

Stack:

  • Ubuntu
  • nginx
  • MariaDB
  • PHP-FPM
  • WordPress (install in a subdirectory)

Install

$ sudo apt-get install nginx

$ sudo apt-get install software-properties-common
$ sudo apt-key adv --recv-keys --keyserver hkp://keyserver.ubuntu.com:80 0xcbcb082a1bb943db
$ sudo add-apt-repository 'deb http://ftp.yz.yamagata-u.ac.jp/pub/dbms/mariadb/repo/10.0/ubuntu trusty main'
$ sudo apt-get update
$ sudo apt-get install mariadb-server

$ sudo apt-get install php5-mysql

$ sudo apt-get install php5-curl php5-fpm

ref:
https://downloads.mariadb.org/mariadb/repositories/

Configuration

PHP

$ sudo vim /etc/php5/fpm/php.ini
# do these
cgi.fix_pathinfo=0
upload_max_filesize = 100M
post_max_size=100M

$ sudo vim /etc/php5/fpm/pool.d/www.conf
# do this
listen = /var/run/php5-fpm.sock

$ sudo service php5-fpm restart

nginx

$ sudo rm /etc/nginx/sites-enabled/default

$ sudo vim /etc/nginx/nginx.conf
# do these
worker_processes 2;
client_max_body_size 100m;

Database

$ mysql -u root -p

mysql>
CREATE DATABASE vinta_blog;
CREATE USER [email protected];
SET PASSWORD FOR [email protected]= PASSWORD("YOUR_PASSOWRD");
GRANT ALL PRIVILEGES ON vinta_blog.* TO [email protected] IDENTIFIED BY 'YOUR_PASSOWRD';
FLUSH PRIVILEGES;

WordPress

$ mkdir ~/wordpress
$ wget http://wordpress.org/latest.zip
$ unzip latest.zip
$ mv latest blog
$ mkdir backup-db

$ chmod 777 backup-db
$ sudo chown -R www-data:www-data wordpress
$ sudo usermod -a -G www-data vinta

ref:
https://www.digitalocean.com/community/articles/how-to-install-linux-nginx-mysql-php-lemp-stack-on-ubuntu-12-04
https://www.digitalocean.com/community/articles/how-to-install-wordpress-with-nginx-on-ubuntu-12-04
https://www.digitalocean.com/community/articles/how-to-configure-single-and-multiple-wordpress-site-settings-with-nginx