Create a custom migration in Django

Create a custom migration in Django

# create an empty migration file
$ ./manage.py makemigrations --empty --name convert_to_utf8mb4 your_app

in your_app/migrations/0002_convert_to_utf8mb4.py

from __future__ import unicode_literals

from django.db import migrations

class Migration(migrations.Migration):

    dependencies = [
        ('your_app', '0001_initial'),
    ]

    operations = [
        migrations.RunSQL(
            'ALTER TABLE app_repostarring CHANGE repo_description repo_description VARCHAR(191) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;'
        ),
    ]

ref:
https://docs.djangoproject.com/en/dev/ref/migration-operations/#runsql

Lazy evaluation in Django middlewares

Lazy evaluation in Django middlewares

Attach a lazy evaluated function as a property of request in a middleware.

from django.contrib.gis.geoip import GeoIP
from django.utils.functional import SimpleLazyObject

def get_country_code(request):
    g = GeoIP()
    location = g.country(request.META['REMOTE_ADDR'])
    country_code = location.get('country_code', 'TW')

    return country_code

class CountryAndSiteMiddleware(object):

    def process_request(self, request):
        request.COUNTRY_CODE = SimpleLazyObject(lambda: get_country_code(request))

Then you could use request.COUNTRY_CODE whenever you want.

Django's get_or_create() may raise IntegrityError but subsequent get() raises DoesNotExist

Django's get_or_create() may raise IntegrityError but subsequent get() raises DoesNotExist

Django 的 SomeModel.objects.get_or_create() 實際上的步驟是:

  1. get
  2. if problem: save + some_trickery
  3. if still problem: get again
  4. if still problem: surrender and raise

get_or_create() 拋出 IntegrityError 但是接著 get() 一次卻會是 DoesNotExist,這個錯誤常常會發生在 MySQL 的 AUTOCOMMIT=OFF 且 isolation level 是 REPEATABLE READ 的情況下。MySQL 默認採用 REPEATABLE READ(而 PostgreSQL 預設是 READ COMMITTED),而 REPEATABLE READ 的定義是在同一個 transaction 之內 執行相同的 SELECT 一定會拿到相同的結果。

with transaction.commit_on_success():
    try:
        user, created = User.objects.get_or_create(id=self.id, username=self.username)
    except IntegrityError:
        # 這個 get() 可能會發生 DoesNotExist
        user = User.objects.get(id=self.id)

會發生那個 DoesNotExist 通常是因為:

  1. thread a 執行 get(),但是 DoesNotExist
  2. 所以 thread a 接著 create(),但是在資料建立之前
  3. thread b 幾乎在同一個時間點也執行 get(),也是 DoesNotExist
  4. thread a 成功地 create()
  5. 因為在 3 拿不到資料,所以 thread b 也執行 create(),但是因為 UNIQUE CONSTRAINT,導致失敗了
  6. 所以 thread b 又會 get() 一次,但是因為 REPEATABLE READ,thread b 不會拿到 thread a 建立的資料,所以 DoesNotExist
  7. 所以那個 User.DoesNotExist 就是 thread b 拋出來的錯誤
  8. thread a 是成功的,你事後去資料庫裡查詢,會看到那個 User 的資料好端端地在那裡

ref:
http://stackoverflow.com/questions/2235318/how-do-i-deal-with-this-race-condition-in-django
https://blog.ionelmc.ro/2014/12/28/terrible-choices-mysql/

解決的方法,除了就乾脆不要在同一個 transaction 裡之外,基本上就只能把 isolation level 改成 READ COMMITTED 了。似乎沒有其他辦法了,因為這就是 isolation level 的定義。

As of MariaDB/MySQL 5.1, if you use READ COMMITTED or enable innodb_locks_unsafe_for_binlog, you must use row-based binary logging.

SHOW VARIABLES LIKE 'binlog_format';
SET SESSION binlog_format = 'ROW';
SET GLOBAL binlog_format = 'ROW';

ref:
http://dba.stackexchange.com/questions/2678/how-do-i-show-the-binlog-format-on-a-mysql-server

Django 和 Celery 都建議使用 READ COMMITTED
https://docs.djangoproject.com/en/dev/ref/models/querysets/#get-or-create
https://code.djangoproject.com/ticket/13906
https://code.djangoproject.com/ticket/6641
http://docs.celeryproject.org/en/latest/faq.html#mysql-is-throwing-deadlock-errors-what-can-i-do

方法一

SELECT @@GLOBAL.tx_isolation, @@tx_isolation;

# MySQL
SET SESSION tx_isolation='READ-COMMITTED';
SET GLOBAL tx_isolation='READ-COMMITTED';

# MariaDB
SET SESSION TRANSACTION ISOLATION LEVEL READ COMMITTED;
SET GLOBAL TRANSACTION ISOLATION LEVEL READ COMMITTED;

ref:
http://dev.mysql.com/doc/refman/5.5/en/set-transaction.html

方法二

/etc/mysql/my.cnf

[mysqld]
transaction-isolation = READ-COMMITTED

方法三

in settings.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.mysql',
        ...
        'OPTIONS': {
            'init_command': 'SET SESSION tx_isolation="READ-COMMITTED"',
        },
    }
}

方法四

自己寫一個 get_or_create()

from django.db import transaction

@transaction.atomic
# or
@transaction.commit_on_success
def my_get_or_create(...):
    try:
        obj = MyObj.objects.create(...)
    except IntegrityError:
        transaction.commit()
        obj = MyObj.objects.get(...)
    return obj

ref:
http://stackoverflow.com/questions/2235318/how-do-i-deal-with-this-race-condition-in-django

MySQL 默認是 AUTOCOMMIT
http://dev.mysql.com/doc/refman/5.5/en/commit.html

Django 默認也是 AUTOCOMMIT,除非你使用了 TransactionMiddleware(Django 1.5 以前)或是設置 ATOMIC_REQUESTS = False(Django 1.6 以後)。TransactionMiddleware 的效果類似 @transaction.commit_on_success@transaction.atomic,都是把 AUTOCOMMIT 關掉,差別在於範圍不同。

ref:
http://django.readthedocs.io/en/1.5.x/topics/db/transactions.html#django-s-default-transaction-behavior
http://django.readthedocs.io/en/1.6.x/topics/db/transactions.html#managing-autocommit
http://django.readthedocs.io/en/1.6.x/ref/settings.html#std:setting-DATABASE-ATOMIC_REQUESTS

Configure i18n translation in Django

Configure i18n translation in Django

Internationalization and localization for your Django project.

Configuration

in settings.py

USE_I18N = True

# Django 1.6 要用 'zh-tw' 或 'zh-cn'
# Django 1.7+ 要用 'zh-hant' 或 'zh-hans'
LANGUAGE_CODE = 'zh-hant'

LANGUAGES = (
    ('en', 'English'),
    ('zh-hans', 'Simplified Chinese'),
    ('zh-hant', 'Traditional Chinese'),
)

LOCALE_PATHS = (
    os.path.join(BASE_DIR, 'locale/'),
)

# 要記得加上 LocaleMiddleware,才會針對 request headers 使用對應的語言
MIDDLEWARE_CLASSES = (
    ...
    'django.middleware.locale.LocaleMiddleware',
    ...
)

ref:
https://docs.djangoproject.com/en/dev/ref/settings/#languages
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#how-django-discovers-language-preference

in urls.py

urlpatterns = patterns('',
    ...
    url(r'^i18n/', include('django.conf.urls.i18n')),
    ...
)

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#the-set-language-redirect-view

Usage

在 models.py 中通常都會使用 ugettext_lazy() 而不是 ugettext()。因為 gettext_lazy() 其中的值是在被訪問的時候才翻譯,而不是在呼叫 gettext_lazy() 的時候就翻譯。另外,ugettext()ugettext_lazy() 出來的字串都是 unicode。pgettext() 的作用跟 ugettext() 一樣,只是多了一個參數可以傳 context 進去。

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#lazy-translation
https://docs.djangoproject.com/en/dev/ref/utils/#django.utils.translation.pgettext

要在 ugettext() 中使用 string format,必須要用以下的形式:

msg = _(u'您有 %(not_ready_count)s 項行程的尚未填寫「司機備忘錄」。') % {'not_ready_count': not_ready_count}

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#internationalization-in-python-code

in base.html

{% load i18n %}

<!DOCTYPE html>
<html>
  <head>
    <meta charset="utf-8">
    <title>Heelstagram</title>
  </head>
  <body>
    {% block content %}{% endblock %}

    <form action="/i18n/setlang/" method="POST">
      {% csrf_token %}
      <input name="next" type="hidden" value="{{ redirect_to }}" />
      <select name="language">
      {% get_language_info_list for LANGUAGES as LANGUAGES %}
      {% for language in LANGUAGES %}
        <option value="{{ language.code }}">{{ language.name_local }} ({{ language.code }})</option>
      {% endfor %}
      </select>
      <input type="submit" value="{% trans 'Change Language' %}" />
    </form>
  </body>
</html>

in home.html

{% extends 'base.html' %}

{# 即使 base.html 已經 load i18n 了,在每個 template 還是得在 load 一次 #}
{% load i18n %}

{% block content %}
<p>current language: "{{ LANGUAGE_CODE }}"</p>

{# 在 .po 中會表示為 msgid "string 1" #}
<p>1: {{ VAR_1 }} 會被翻譯</p>

{# 這兩行是等價的,在 .po 中會表示為 msgid "string 3" #}
<p>2: {% trans VAR_3 %} 會被翻譯</p>
<p>3: {% trans "string 3" %} 會被翻譯</p>

{# 在 .po 中會表示為: #}
{# msgctxt "說明的文字" #}
{# msgid "User" #}
<p>4: {% trans 'User' context '說明的文字' %} 會被翻譯</p>

<p>5: {{ VAR_3 }} 不會被翻譯</p>

{# 在 .po 中會表示為 msgid "%(VAR_2)s 這整句會被翻譯,包含 VAR_2 的值" #}
{# VAR_2 也會被翻譯是因為它等於 _('string 2') #}
{# blocktrans 中的變數不能是 artist.name 這種形式,必須用 artist_name 或是 with #}
<p>6: {% blocktrans %}{{ VAR_2 }} 這整句會被翻譯,包含 VAR_2 的值{% endblocktrans %}</p>

{# VAR_4 不會被翻譯是因為它等於 'string 4',只是一個單純的字串 #}
<p>7: {% blocktrans %}{{ VAR_4 }} 這整句會被翻譯,但是不包含 VAR_4 的值{% endblocktrans %}</p>
{% endblock %}

blocktrans 的作用是讓你在翻譯字串中插入 template contexts 變數。

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#trans-template-tag
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#internationalization-in-template-code

Create Language Files

要先安裝 gettext,否則會出現 /bin/sh: xgettext: command not found 的錯誤

# 必須手動建立 locale 目錄(放在 project 或 app 的根目錄)
$ mkdir -p locale

# 第一次執行,必須指定 locale name,不能直接用 `-a` 參數,這樣才會在 locale 底下產生相對應的目錄
# 注意!是 zh_TW 而不是 zh-tw
# 在專案根目錄執行 makemessages 就是對整個專案的所有 apps 產生翻譯檔案
$ ./manage.py makemessages -l en
$ ./manage.py makemessages -l ja
$ ./manage.py makemessages -l zh_CN
$ ./manage.py makemessages -l zh_TW
$ ./manage.py makemessages -l zh_Hans
$ ./manage.py makemessages -l zh_Hant
$ ./manage.py makemessages --all

# .js 的翻譯要額外指定 `-d djangojs` 才會產生 djangojs.po
$ ./manage.py makemessages -l zh_Hant -d djangojs --ignore=node_modules

# 如果你的 django.po 裡有些字串被標記為 `#, fuzzy` 的話,要記得刪掉,否則該字串不會被翻譯
$ ./manage.py compilemessages

# 只對特定 app 產生翻譯檔案
$ cd your_app
$ django-admin.py makemessages -l zh_Hant
$ django-admin.py compilemessages

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#localization-how-to-create-language-files
https://docs.djangoproject.com/en/dev/ref/django-admin/#makemessages

每次 compilemessages 完要記得重啟 server

How Django discovers translations

django-admin.py makemessages -a 這個指令會去收集整個 project 裡的所有 apps 的 locale 字串。
優先權最高的是 LOCALE_PATHS 定義的那個目錄,找不到的話才會去找個別 app 之下的 locale 目錄。

ref:
https://docs.djangoproject.com/en/dev/topics/i18n/translation/#how-django-discovers-translations

強制使用某種語言

除了可以用上面那個 /i18n/setlang/ 的 form 表單之外,也可以在 views 裡面這樣寫:

from django.shortcuts import render_to_response
from django.template import RequestContext
from django.utils.translation import activate
from django.utils.translation import ugettext as _

def common_processor(request):
    contexts = {
        'T1': _('string 1'),
    }

    return contexts

def home(request):
    activate('zh-tw')  # 強制使用正體中文,覆蓋掉 user 和 browser 的設定

    contexts = {
        'T2': _('string 2'),
        'T3': 'string 3',
    }

    return render_to_response('home.html', contexts, RequestContext(request, processors=[common_processor]))

ref:
https://docs.djangoproject.com/en/dev/ref/utils/#module-django.utils.translation

JavaScript

in urls.py

urlpatterns = patterns(
    '',
    ...
    url(r'^jsi18n/$', 'django.views.i18n.javascript_catalog'),
    ...
)

in your_shit.html

<script src="{% url 'django.views.i18n.javascript_catalog' %}"></script>
<script>
    var text = gettext('要被翻譯的字串');
</script>

gettext() 而不是 ugettext()

# 要加上 -d djangojs 才會去 parse .js 裡的字串
$ ./manage.py makemessages --all -d djangojs
Tools for Profiling your Python Projects

Tools for Profiling your Python Projects

The first aim of profiling is to test a representative system to identify what's slow, using too much RAM, causing too much disk I/O or network I/O. You should keep in mind that profiling typically adds an overhead to your code.

In this post, I will introduce tools you could use to profile your Python or Django projects, including: timer, pycallgraph, cProfile, line-profiler, memory-profiler.

ref:
https://stackoverflow.com/questions/582336/how-can-you-profile-a-script
https://www.airpair.com/python/posts/optimizing-python-code

timer

The simplest way to profile a piece of code.

ref:
https://docs.python.org/3/library/timeit.html

pycallgraph

pycallgraph is a Python module that creates call graph visualizations for Python applications.

ref:
https://pycallgraph.readthedocs.org/en/latest/

$ sudo apt-get install graphviz
$ pip install pycallgraph
# in your_app/middlewares.py
from pycallgraph import Config
from pycallgraph import PyCallGraph
from pycallgraph.globbing_filter import GlobbingFilter
from pycallgraph.output import GraphvizOutput
import time

class PyCallGraphMiddleware(object):

    def process_view(self, request, callback, callback_args, callback_kwargs):
        if 'graph' in request.GET:
            config = Config()
            config.trace_filter = GlobbingFilter(include=['rest_framework.*', 'api.*', 'music.*'])
            graphviz = GraphvizOutput(output_file='pycallgraph-{}.png'.format(time.time()))
            pycallgraph = PyCallGraph(output=graphviz, config=config)
            pycallgraph.start()

            self.pycallgraph = pycallgraph

    def process_response(self, request, response):
        if 'graph' in request.GET:
            self.pycallgraph.done()

        return response
# in settings.py
MIDDLEWARE_CLASSES = (
    'your_app.middlewares.PyCallGraphMiddleware',
    ...
)
$ python manage.py runserver 0.0.0.0:8000
$ open http://127.0.0.1:8000/your_endpoint/?graph=true

cProfile

cProfile is a tool in Python's standard library to understand which functions in your code take the longest to run. It will give you a high-level view of the performance problem so you can direct your attention to the critical functions.

ref:
http://igor.kupczynski.info/2015/01/16/profiling-python-scripts.html
https://ymichael.com/2014/03/08/profiling-python-with-cprofile.html

$ python -m cProfile manage.py test member
$ python -m cProfile -o my-profile-data.out manage.py test --failtest
$ python -m cProfile -o my-profile-data.out manage.py runserver 0.0.0.0:8000

$ pip install cprofilev
$ cprofilev -f my-profile-data.out -a 0.0.0.0 -p 4000
$ open http://127.0.0.1:4000

cProfile with django-cprofile-middleware

$ pip install django-cprofile-middleware
# in settings.py
MIDDLEWARE_CLASSES = (
    ...
    'django_cprofile_middleware.middleware.ProfilerMiddleware',
)

Open any url with a ?prof suffix to do the profiling, for instance, http://localhost:8000/foo/?prof

ref:
https://github.com/omarish/django-cprofile-middleware

cProfile with django-extension and kcachegrind

kcachegrind is a profiling data visualization tool, used to determine the most time consuming execution parts of a program.

ref:
http://django-extensions.readthedocs.org/en/latest/runprofileserver.html

$ pip install django-extensions
# in settings.py
INSTALLED_APPS += (
    'django_extensions',
)
$ mkdir -p my-profile-data

$ python manage.py runprofileserver \
--noreload \
--nomedia \
--nostatic \
--kcachegrind \
--prof-path=my-profile-data \
0.0.0.0:8000

$ brew install qcachegrind --with-graphviz
$ qcachegrind my-profile-data/root.003563ms.1441992439.prof
# or
$ sudo apt-get install kcachegrind
$ kcachegrind my-profile-data/root.003563ms.1441992439.prof

cProfile with django-debug-toolbar

You're only able to use django-debug-toolbar if your view returns HTML, it needs a place to inject the debug panels into your DOM on the webpage.

ref:
https://github.com/django-debug-toolbar/django-debug-toolbar

$ pip install django-debug-toolbar
# in settiangs.py
INSTALLED_APPS += (
    'debug_toolbar',
)

DEBUG_TOOLBAR_PANELS = [
    ...
    'debug_toolbar.panels.profiling.ProfilingPanel',
    ...
]

line-profiler

line-profiler is a module for doing line-by-line profiling of functions. One of my favorite tools.

ref:
https://github.com/rkern/line_profiler

$ pip install line-profiler
# in your_app/views.py
def do_line_profiler(view=None, extra_view=None):
    import line_profiler

    def wrapper(view):
        def wrapped(*args, **kwargs):
            prof = line_profiler.LineProfiler()
            prof.add_function(view)
            if extra_view:
                [prof.add_function(v) for v in extra_view]
            with prof:
                resp = view(*args, **kwargs)
            prof.print_stats()
            return resp

        return wrapped

    if view:
        return wrapper(view)

    return wrapper

@do_line_profiler
def your_view(request):
    pass

ref:
https://djangosnippets.org/snippets/10483/

There is a pure Python alternative: pprofile.
https://github.com/vpelletier/pprofile

line-profiler with django-devserver

ref:
https://github.com/dcramer/django-devserver

$ pip install git+git://github.com/dcramer/django-devserver#egg=django-devserver

in settings.py

INSTALLED_APPS += (
    'devserver',
)

DEVSERVER_MODULES = (
    ...
    'devserver.modules.profile.LineProfilerModule',
    ...
)

DEVSERVER_AUTO_PROFILE = False

in your_app/views.py

from devserver.modules.profile import devserver_profile

@devserver_profile()
def your_view(request):
    pass

line-profiler with django-debug-toolbar-line-profiler

ref:
http://django-debug-toolbar.readthedocs.org/en/latest/
https://github.com/dmclain/django-debug-toolbar-line-profiler

$ pip install django-debug-toolbar django-debug-toolbar-line-profiler
# in settings.py
INSTALLED_APPS += (
    'debug_toolbar',
    'debug_toolbar_line_profiler',
)

DEBUG_TOOLBAR_PANELS = [
    ...
    'debug_toolbar_line_profiler.panel.ProfilingPanel',
    ...
]

memory-profiler

This is a Python module for monitoring memory consumption of a process as well as line-by-line analysis of memory consumption for Python programs.

ref:
https://pypi.python.org/pypi/memory_profiler

$ pip install memory-profiler psutil
# in your_app/views.py
from memory_profiler import profile

@profile(precision=4)
def your_view(request):
    pass

There are other options:
http://stackoverflow.com/questions/110259/which-python-memory-profiler-is-recommended

dogslow

ref:
https://bitbucket.org/evzijst/dogslow

django-slow-tests

ref:
https://github.com/realpython/django-slow-tests