Transaction isolation levels in MySQL

Transaction isolation levels in MySQL

Dirty reads 髒讀

Transaction 1 讀到了 Transaction 2 的未提交的值
READ COMMITED 可以避免這個問題
READ UNCOMMITTED 會有這個問題

A dirty read occurs when a transaction is allowed to read data from a row that has been modified by another running transaction and not yet committed.

Non-repeatable reads 不可重複讀

在同一個 Transaction 裡 不同的時間點 讀取了同一個 row 卻得到了不同的值(但是可能使用了不同的 SELECT 語句去讀到同一個 row)
REPEATABLE READ 可以避免這個問題
READ COMMITED 會有這個問題

A non-repeatable read occurs, when during the course of a transaction, a row is retrieved twice and the values within the row differ between reads.

When Transaction 2 commits successfully, which means that its changes to the row with id 1 should become visible. However, Transaction 1 has already seen a different value for age in that row. At the SERIALIZABLE and REPEATABLE READ isolation levels, the DBMS must return the old value for the second SELECT. At READ COMMITTED and READ UNCOMMITTED, the DBMS may return the updated value; this is a non-repeatable read.

Phantom reads 幻讀

有點類似 Non-repeatable reads
在同一個 Transaction 裡 不同的時間點 兩個相同但是先後執行的 SQL 卻得到了不同的結果
差別在於 Phantom reads 專門指沒有用 range locks 時
先後兩個同樣的 SELECT 卻返回了不同數量的結果
SERIALIZABLE 可以避免以上的三種問題

也可以說 non-repeatable reads 的原因是 UPDATE
而 phantom reads 的原因則是 INSERT 和 DELETE

A phantom read occurs when, in the course of a transaction, two identical queries are executed, and the collection of rows returned by the second query is different from the first.

READ UNCOMMITTED

允許 dirty reads
可能會讀到其他 transaction 尚未提交的數據

READ COMMITED

不會有 dirty reads
只會讀到其他 transaction 已經提交的數據
但是允許 non-repeatable reads

大部份資料庫默認的 isolation level 都是 READ COMMITED

REPEATABLE READ

不會有 dirty reads 和 non-repeatable reads
但是允許 phantom reads

MySQL 默認的 isolation level 是 REPEATABLE READ
但是 InnoDB 透過 next-key lock 也防止了 phantom reads

SERIALIZABLE

不會有 dirty reads、non-repeatable reads 和 phantom reads
讀寫互斥
所以讀和寫都只能一個一個依序執行

ref:
https://www.percona.com/blog/2012/08/28/differences-between-read-committed-and-repeatable-read-transaction-isolation-levels/

Configure i18n translation in Django

Configure i18n translation in Django

Internationalization and localization for your Django project.

Configuration

in settings.py

USE_I18N = True

# Django 1.6 要用 'zh-tw' 或 'zh-cn'
# Django 1.7+ 要用 'zh-hant' 或 'zh-hans'
LANGUAGE_CODE = 'zh-hant'

LANGUAGES = (
    ('en', 'English'),
    ('zh-hans', 'Simplified Chinese'),
    ('zh-hant', 'Traditional Chinese'),
)

LOCALE_PATHS = (
    os.path.join(BASE_DIR, 'locale/'),
)

# 要記得加上 LocaleMiddleware,才會針對 request headers 使用對應的語言
MIDDLEWARE_CLASSES = (
    ...
    'django.middleware.locale.LocaleMiddleware',
    ...
)

ref:
https://docs.djangoproject.com/en/dev/ref/settings/#languages
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#how-django-discovers-language-preference

in urls.py

urlpatterns = patterns('',
    ...
    url(r'^i18n/', include('django.conf.urls.i18n')),
    ...
)

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#the-set-language-redirect-view

Usage

在 models.py 中通常都會使用 ugettext_lazy() 而不是 ugettext()。因為 gettext_lazy() 其中的值是在被訪問的時候才翻譯,而不是在呼叫 gettext_lazy() 的時候就翻譯。另外,ugettext()ugettext_lazy() 出來的字串都是 unicode。pgettext() 的作用跟 ugettext() 一樣,只是多了一個參數可以傳 context 進去。

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#lazy-translation
https://docs.djangoproject.com/en/dev/ref/utils/#django.utils.translation.pgettext

要在 ugettext() 中使用 string format,必須要用以下的形式:

msg = _(u'您有 %(not_ready_count)s 項行程的尚未填寫「司機備忘錄」。') % {'not_ready_count': not_ready_count}

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#internationalization-in-python-code

in base.html

{% load i18n %}

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Heelstagram</title>
  </head>
  <body>
    {% block content %}{% endblock %}

    <form action="/i18n/setlang/" method="POST">
      {% csrf_token %}
      <input name="next" type="hidden" value="{{ redirect_to }}" />
      <select name="language">
      {% get_language_info_list for LANGUAGES as LANGUAGES %}
      {% for language in LANGUAGES %}
        <option value="{{ language.code }}">{{ language.name_local }} ({{ language.code }})</option>
      {% endfor %}
      </select>
      <input type="submit" value="{% trans 'Change Language' %}" />
    </form>
  </body>
</html>

in home.html

{% extends 'base.html' %}

{# 即使 base.html 已經 load i18n 了,在每個 template 還是得在 load 一次 #}
{% load i18n %}

{% block content %}
<p>current language: "{{ LANGUAGE_CODE }}"</p>

{# 在 .po 中會表示為 msgid "string 1" #}
<p>1: {{ VAR_1 }} 會被翻譯</p>

{# 這兩行是等價的,在 .po 中會表示為 msgid "string 3" #}
<p>2: {% trans VAR_3 %} 會被翻譯</p>
<p>3: {% trans "string 3" %} 會被翻譯</p>

{# 在 .po 中會表示為: #}
{# msgctxt "說明的文字" #}
{# msgid "User" #}
<p>4: {% trans 'User' context '說明的文字' %} 會被翻譯</p>

<p>5: {{ VAR_3 }} 不會被翻譯</p>

{# 在 .po 中會表示為 msgid "%(VAR_2)s 這整句會被翻譯,包含 VAR_2 的值" #}
{# VAR_2 也會被翻譯是因為它等於 _('string 2') #}
{# blocktrans 中的變數不能是 artist.name 這種形式,必須用 artist_name 或是 with #}
<p>6: {% blocktrans %}{{ VAR_2 }} 這整句會被翻譯,包含 VAR_2 的值{% endblocktrans %}</p>

{# VAR_4 不會被翻譯是因為它等於 'string 4',只是一個單純的字串 #}
<p>7: {% blocktrans %}{{ VAR_4 }} 這整句會被翻譯,但是不包含 VAR_4 的值{% endblocktrans %}</p>
{% endblock %}

blocktrans 的作用是讓你在翻譯字串中插入 template contexts 變數。

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#trans-template-tag
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#internationalization-in-template-code

Create Language Files

要先安裝 gettext,否則會出現 /bin/sh: xgettext: command not found 的錯誤

# 必須手動建立 locale 目錄(放在 project 或 app 的根目錄)
$ mkdir -p locale

# 第一次執行,必須指定 locale name,不能直接用 `-a` 參數,這樣才會在 locale 底下產生相對應的目錄
# 注意!是 zh_TW 而不是 zh-tw
# 在專案根目錄執行 makemessages 就是對整個專案的所有 apps 產生翻譯檔案
$ ./manage.py makemessages -l en
$ ./manage.py makemessages -l ja
$ ./manage.py makemessages -l zh_CN
$ ./manage.py makemessages -l zh_TW
$ ./manage.py makemessages -l zh_Hans
$ ./manage.py makemessages -l zh_Hant
$ ./manage.py makemessages --all

# .js 的翻譯要額外指定 `-d djangojs` 才會產生 djangojs.po
$ ./manage.py makemessages -l zh_Hant -d djangojs --ignore=node_modules

# 如果你的 django.po 裡有些字串被標記為 `#, fuzzy` 的話,要記得刪掉,否則該字串不會被翻譯
$ ./manage.py compilemessages

# 只對特定 app 產生翻譯檔案
$ cd your_app
$ django-admin.py makemessages -l zh_Hant
$ django-admin.py compilemessages

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#localization-how-to-create-language-files
https://docs.djangoproject.com/en/dev/ref/django-admin/#makemessages

每次 compilemessages 完要記得重啟 server

How Django discovers translations

django-admin.py makemessages -a 這個指令會去收集整個 project 裡的所有 apps 的 locale 字串。
優先權最高的是 LOCALE_PATHS 定義的那個目錄,找不到的話才會去找個別 app 之下的 locale 目錄。

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#how-django-discovers-translations

強制使用某種語言

除了可以用上面那個 /i18n/setlang/ 的 form 表單之外,也可以在 views 裡面這樣寫:

from django.shortcuts import render_to_response
from django.template import RequestContext
from django.utils.translation import activate
from django.utils.translation import ugettext as _

def common_processor(request):
    contexts = {
        'T1': _('string 1'),
    }

    return contexts

def home(request):
    activate('zh-tw')  # 強制使用正體中文,覆蓋掉 user 和 browser 的設定

    contexts = {
        'T2': _('string 2'),
        'T3': 'string 3',
    }

    return render_to_response('home.html', contexts, RequestContext(request, processors=[common_processor]))

ref:
https://docs.djangoproject.com/en/dev/ref/utils/#module-django.utils.translation

JavaScript

in urls.py

urlpatterns = patterns(
    '',
    ...
    url(r'^jsi18n/$', 'django.views.i18n.javascript_catalog'),
    ...
)

in your_shit.html

<script src="{% url 'django.views.i18n.javascript_catalog' %}"></script>
<script>
    var text = gettext('要被翻譯的字串');
</script>

gettext() 而不是 ugettext()

# 要加上 -d djangojs 才會去 parse .js 裡的字串
$ ./manage.py makemessages --all -d djangojs
Python is call-by-assignment

Python is call-by-assignment

If you pass a mutable object into a method, the method gets a reference to that same object and you can mutate it to your heart's delight, but if you rebind the reference in the method, the outer scope will know nothing about it, and after you're done, the outer reference will still point at the original object.

If you pass an immutable object to a method, you still can't rebind the outer reference, and you can't even mutate the object.

例如 def foo(bar): 這個 function 的參數傳遞,其實是在 foo 的 local namespace 裡幫傳進去的物件綁定了一個叫做 bar 的名字(所謂的 assignment);如果你在 foo 裡 re-assign 了一個新的物件給 bar,實際上是把 bar 這個名字綁定到那個新的物件。

  • mutable objects: list, dict, set 的行為同 call-by-reference
  • immutable objects: boolean, int, float, long, str, unicode, tuple 的行為同 call-by-value

不過如果你把一個 mutable 物件放進 immutable 物件裡,例如把 list 放進 tuple 裡,則修改了 list 之後,那個 tuple 裡的 list 也是會被修改。

a_list = [1, 2, 3]
a_tuple = (a_list, 'a', 'b', 'c')
a_list.append(4)

Assignment is the binding of a name to an object: name = 'Molly'.
Assignment between names doesn't create a new object, both names are simply bound to the same object: nickname = name.

name = 'Mollie'
name = 'Vinta'

所謂的 assign 這個動作,其實是幫 'Mollie' 這個字串取一個名字叫做 name,所以如果你又加上一句 name = 'Vinta',實際上是建立了一個新的物件(字串 'Vinta'),再把這個新字串綁定到 name 這個名字。

# if bar refers to a mutable object
def foo(bar):
    bar.append('new value')
    print(bar)
    # output: ['old value', 'new value']

answer_list = ['old value', ]
foo(answer_list)
print(answer_list)
# output: ['old value', 'new value']

# if bar refers to an immutable object
def foo(bar):
    bar = 'new value'
    print(bar)
    # output: 'new value'

answer_list = 'old value'
foo(answer_list)
print(answer_list)
# output: 'old value'

# if bar refers to a mutable object and re-assign it in foo
def foo(bar):
    bar = ['new value', ]
    print(bar)
    # output: ['new value', ]

answer_list = ['old value', ]
foo(answer_list)
print(answer_list)
# output: ['old value', ]

ref:
https://stackoverflow.com/questions/986006/how-do-i-pass-a-variable-by-reference
https://jeffknupp.com/blog/2012/11/13/is-python-callbyvalue-or-callbyreference-neither/
https://docs.python.org/3/faq/programming.html#how-do-i-write-a-function-with-output-parameters-call-by-reference

scope

只有 module、class、function 才有建立新的 scope,if、while、for 並不會建立新的 scope。

some_list = [
    {
        'id': 1,
        'is_change': False,
    },
    {
        'id': 2,
        'is_change': False,
    },
    {
        'id': 3,
        'is_change': False,
    },
    {
        'id': 4,
        'is_change': False,
    },
]

for item in some_list:
    item['is_change'] = True

some_list 的每個 item 都會被更新,因為是 mutable objects 的行為是 call-by-reference。所以你不需要這樣:

new_list = []
for item in some_list:
    item['is_change'] = True
    new_list.append(item)
Upload files to Amazon S3 when Travis CI builds pass

Upload files to Amazon S3 when Travis CI builds pass

Assume that you want to upload a xxx.whl file generated by pip wheel to Amazon S3 so that you will be able to run pip install https://url/to/s3/bucket/xxx.whl.

CAUTION! By default, only master branch's builds could trigger deployments in Travis CI.

Configuration

before_install:
  - pip install -U pip
  - pip install wheel

script:
  - python setup.py test

before_deploy:
  - pip wheel --wheel-dir=wheelhouse .

deploy:
  provider: s3
  access_key_id: "YOUR_KEY"
  secret_access_key: "YOUR_SECRET"
  bucket: YOUR_BUCKET
  acl: public_read
  local_dir: wheelhouse
  upload_dir: wheels
  skip_cleanup: true
# install from an URL directly
$ pip install https://url/to/s3/bucket/wheels/xxx.whl

ref:
https://docs.travis-ci.com/user/deployment/s3

Setup a static website on Amazon S3

Setup a static website on Amazon S3

Say that you would like to host your static site on Amazon S3 with a custom domain and, of course, HTTPS.

Create two S3 buckets

To serve requests from both root domain such as codetengu.com and subdomain such as www.codetengu.com, you must create two buckets named exactly codetengu.com and www.codetengu.com.

In this post, I assume that you want to redirect www.codetengu.com to codetengu.com.

ref:
https://docs.aws.amazon.com/AmazonS3/latest/dev/website-hosting-custom-domain-walkthrough.html

Upload your static files

$ cd /path/to/your_project_root/

$ aws s3 sync . s3://codetengu.com \
--acl "public-read" \
--exclude "*.DS_Store" \
--exclude "*.gitignore" \
--exclude ".git/*" \
--dryrun

$ aws s3 website s3://codetengu.com --index-document index.html --error-document error.html

ref:
https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html

Setup bucket policy for public accessing

In your S3 Management Console, click codetengu.com bucket > Properties > Edit bucket policy, enter:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AddPerm",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::codetengu.com/*"
        }
    ]
}

Setup www redirecting

In your S3 Management Console, click www.codetengu.com bucket > Properties > Static Website Hosting, choose Redirect all requests to another host name, type codetengu.com.

Now you're able to access your website via:

Configure a custom domain

In the "Setting Up a Static Website Using a Custom Domain" guide I mentioned above, it uses Amazon Route 53 to manage DNS records; In this post, I use CloudFlare as my website's DNS provider instead.

  • Create a CNAME for codetengu.com to point to codetengu.com.s3-website-ap-northeast-1.amazonaws.com
  • Create a CNAME for www.codetengu.com to point to codetengu.com.s3-website-ap-northeast-1.amazonaws.com

Yep, you CAN create a CNAME record for root domain on CloudFlare, just like your can add an "Alias" on Route 53.

Wait for the DNS records to propagate then visit https://codetengu.com/.