Celery 3.1 with Django

Install

$ pip install celery

# for Amazon SQS
$ pip install boto

# for IronMQ
$ pip install iron-mq iron_celery==0.4.0

ref:
http://docs.celeryproject.org/

Configuration

in init.py(跟 settings.py 放在一起)

from __future__ import absolute_import

# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app

in celery.py(跟 settings.py 放在一起)

from __future__ import absolute_import
import os

from celery import Celery

from django.conf import settings

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'dps.settings')

app = Celery('dps')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

in settings.py

ref:
http://docs.celeryproject.org/en/latest/getting-started/brokers/sqs.html#broker-sqs
https://github.com/iron-io/iron_celery

# for Amazon SQS
AWS_ACCESS_KEY_ID_ESCAPED = urllib.quote_plus(AWS_ACCESS_KEY_ID)
AWS_SECRET_ACCESS_KEY_ESCAPED = urllib.quote_plus(AWS_SECRET_ACCESS_KEY)
BROKER_URL = 'sqs://%s:%[email protected]' % (AWS_ACCESS_KEY_ID_ESCAPED, AWS_SECRET_ACCESS_KEY_ESCAPED)
BROKER_TRANSPORT_OPTIONS = {
    'region': 'ap-northeast-1',
    'queue_name_prefix': 'dps-celery-',
}

# for IronMQ
BROKER_URL = 'ironmq://project_id:[email protected]'

CELERYD_CONCURRENCY = 2

CELERY_ACCEPT_CONTENT = ['json', ]
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'

Usage

in your_app/tasks.py

from __future__ import absolute_import

from celery import shared_task


@shared_task
def add(x, y):
    return x + y

Commands

# 在 project root 執行這個指令即可
# goeasytaxi 就是 celery app 的名稱
$ celery -A goeasytaxi worker -l info

Upstart

in /etc/init/celeryd.conf

description "Celery"

start on runlevel [2345]
stop on runlevel [!2345]

script
    chdir /home/vinta/goeasytaxi
    exec /home/vinta/.virtualenvs/goeasytaxi/bin/celery -A goeasytaxi worker -l info
end script

respawn
$ sudo service celeryd restart

Docker port forwarding and share volume in Mac OS X

2014.10.17 UPDATED:

Docker 1.3 已經解決了 mount volume 和 port forwarding 的問題
你可以直接用 -v /Users/vinta/path/in/mac:/path/in/container-p 8000:8000

不過要注意的是
前一個 8000 是 boot2docker VM 的 port
你可以直接連 192.168.59.103:8000
或是把 boot2docker VM 的 port 再導到 Mac OS X 來

$ VBoxManage controlvm boot2docker-vm natpf1 "Rule 8000,tcp,127.0.0.1,8000,,8000"

--

2014.09.17 UPDATED:

在 Docker 原生地支援 Mac 之前
只能這麼幹了

解決:
在 Mac 不能 mount volume 的問題
在 Mac 不能 port forwarding 的問題

用以下的方式不需要重啟 boot2docker
如果你使用 VirtualBox 的 GUI 界面去修改的話
則要重啟 boot2docker

$ wget http://static.dockerfiles.io/boot2docker-v1.2.0-virtualbox-guest-additions-v4.3.14.iso
$ mv ~/.boot2docker/boot2docker.iso ~/.boot2docker/boot2docker.iso.backup
$ mv boot2docker-v1.2.0-virtualbox-guest-additions-v4.3.14.iso ~/.boot2docker/boot2docker.iso

$ boot2docker init

# 開 volume
# 不過這種方式還是有個問題,就是對 container 來說,那個 shared folder 的權限都會是 root
# 就算 sudo chown -R mysql:mysql /var/lib/mysql/ 也沒有作用
$ VBoxManage sharedfolder add boot2docker-vm -name home -hostpath /Users
$ boot2docker up

# SSH 進去檢查一下
$ boot2docker ssh "ls /Users"

# now you can do something like this
$ docker run -it -v /Users/vinta/your_app:/app ubuntu:14.04 bash

# 開 port
# 第一個 8000 是 host port,即 Mac os x
# 第二個 8000 則是 guest port,即 boot2docker-vm
$ VBoxManage controlvm boot2docker-vm natpf1 "Rule 8000,tcp,127.0.0.1,8000,,8000"
$ VBoxManage controlvm boot2docker-vm natpf1 "Rule 8004,tcp,127.0.0.1,8004,,8004"
$ VBoxManage controlvm boot2docker-vm natpf1 "Rule 8888,tcp,127.0.0.1,8888,,8888"
$ VBoxManage controlvm boot2docker-vm natpf1 "Rule 6379,tcp,127.0.0.1,6379,,6379"
$ VBoxManage controlvm boot2docker-vm natpf1 "Rule 3306,tcp,127.0.0.1,3306,,3306"

# now you can do something like this
$ docker run -it -p 8000:8000 ubuntu:14.04 bash

ref:
http://j.mp/1sXl1eq