I Failed the Turing Test

AWS Lambda Cookbook

2016-05-212026-03-17VintaDevOps, JavaScript, Web Development

AWS Lambda is an event-driven service that lets you upload your code and run it on-demand without having your own servers.

ref:
http://aws.amazon.com/lambda/
http://docs.aws.amazon.com/lambda/latest/dg/limits.html

API Gateway 就是 URL routing
Lambda 則是那些 route (endpoint) 對應的 handler
如果你是用 event 或 schedule 的方式呼叫 Lambda function 的話
可以不用 API Gateway

AWS Lambda 有兩種 invocation type
一是 RequestResponse，同步（例如綁定 API Gateway 和你在 Lambda Management Console 操作的時候）
二是 Event，非同步

Runtimes

AWS Lambda supports the following runtime versions:

nodejs (Node v0.10)
nodejs4.3
java
python

ref:
http://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html

Node.js

const aws = require('aws-sdk');

exports.handle = (event, context, callback) => {
  doYourShit();
  callback(null, 'DONE');
};

每個 Lambda function 會接收三個參數 event、context 和 callback

event 是從外部的 input
可能是來自 S3 object event、DynamoDB stream 或是由 API Gateway POST 進來的 JSON payload

context 則會包含當前這個 Lambda fuction 的一些 metadata
例如 context.getRemainingTimeInMillis()

callback 參數只有 Node.js runtime v4.3 才支援
v0.10 的話得用 context.succeed()、context.fail() 和 context.done()
不過誰他媽還在用 Node.js v0.10

ref:
http://docs.aws.amazon.com/lambda/latest/dg/programming-model.html
http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html
http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-context.html
http://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

Calling another Lambda function in a Lambda function.

要注意的是
你的 Lambda function 的 role 得要有 invoke 其他 Lambda function 的權限才行

const util = require('util');

const aws = require('aws-sdk');

const params = {
  FunctionName: 'LambdaBaku_syncIssue',
  InvocationType: 'Event', // means asynchronous execution
  Payload: JSON.stringify({ issue_number: curatedIssue.number }),
};

lambda.invoke(params, (err, data) => {
  if (err) {
    console.log('FAIL', params);
    console.log(util.inspect(err));
  } else {
    console.log(data);
  }
});

ref:
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/Lambda.html
http://stackoverflow.com/questions/31714788/can-an-aws-lambda-function-call-another

完整的程式碼放在 GitHub 上
https://github.com/CodeTengu/lambdabaku

Users and Roles

如果你是用 apex 來管理 Lambda functions 的話
確保你用的 AWS credential (User) 擁有 AWSLambdaFullAccess 和 AWSLambdaRole 這兩個 permissions

以 project 為單位建立 Role 即可
例如 lambdabaku_role
你可以在 IAM Management Console 找到那些你建立的 roles
基本上用 Basic execution role 就夠了
反正之後可以隨時修改 Role 的 permission / policy
Lambda function 屬於哪個 VPC 是額外指定的
跟 Role 沒有關係
也就是說你用 Basic execution role 還是可以支援 VPC

如果想在 Lambda function 裡存取 DynamoDB
要記得在 Role 裡新增對應的設定

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        },
        {
            "Sid": "Stmt1428341300017",
            "Effect": "Allow",
            "Action": [
                "dynamodb:*"
            ],
            "Resource": [
                "arn:aws:dynamodb:ap-northeast-1:004615714446:table/CodeTengu_Preference",
                "arn:aws:dynamodb:ap-northeast-1:004615714446:table/CodeTengu_WeeklyIssue",
                "arn:aws:dynamodb:ap-northeast-1:004615714446:table/CodeTengu_WeeklyPost"
            ]
        }
    ]
}

Scheduled Events

ref:
http://docs.aws.amazon.com/lambda/latest/dg/with-scheduled-events.html

API Gateway

單純一點的話
Security 可以選 Open with access key
然後到 API Gateway 介面的 API Keys 底下新增一組 access key
然後分配一個 API stage 給它

使用的時候在 HTTP header 加上 x-api-key: YOUR_API_KEY 即可

ref:
http://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-api-keys.html

Related Projects

ref:
https://github.com/serverless/serverless
https://github.com/apex/apex
https://github.com/claudiajs/claudia
https://github.com/garnaat/kappa
https://github.com/Miserlou/Zappa
https://github.com/nficano/python-lambda

淺析 serverless 架構與實作
http://abalone0204.github.io/2016/05/22/serverless-simple-crud/

Deploy Lambda Functions via apex

$ curl https://raw.githubusercontent.com/apex/apex/master/install.sh | sh

$ apex deploy
$ apex invoke syncPublishedIssues --logs
$ echo -n '{"issue_number": 43}' | apex invoke syncIssue --logs

ref:
https://github.com/apex/apex
http://apex.run/

AWS DynamoDB Notes

2016-05-192026-03-17VintaDatabase, DevOps, Web Development

AWS DynamoDB is a fully managed key-value store (also document store) NoSQL database as a service provided by Amazon Web Services. Its pricing model is that you only pay for the throughput (read and write) you use instead of the storage usage and the running hours of database instances.

ref:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html
http://www.slideshare.net/AmazonWebServices/design-patterns-using-amazon-dynamodb

Glossary

DynamoDB is schema-less.

table: a table is a collection of items.
item: an item is a collection of attributes (key-value pairs).
attribute: an attribute is similar to a field or column in other databases.
primary key: one or two attributes that can uniquely identify every item in a table.
- partition key (aka hash key): a simple primary key, composed of one attribute.
- partition key and sort key (aka range key): a composite primary key, composed of two attributes.

ref:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html

Global Secondary Index (GSI)

secondary index 指的是除了 primary key 之外的第二組 key
可以有很多組 secondary index
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html

GSI 可以用在是 partition key 或 partition + sort key 的 table
GSI 跟 primary key 一樣可以 simple 或是 composite 的
GSI 可以隨時增減

如果你不需要 strong consistency 或個別 partition 的資料量大於 10GB
那就用 GSI

ref:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html
http://iamgarlic.blogspot.tw/2015/01/amazon-dynamodb-global-secondary-index.html

Local Secondary Index (LSI)

LSI 只能用在是 partition + sort key 的 table
LSI 必須用原本的 partition key 搭配其他 attribute 做為新的 partition + sort key（LSI 只會是 composite 的）
LSI 只能在建立 table 的時候定義

ref:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LSI.html
http://iamgarlic.blogspot.tw/2015/01/amazon-dynamodb-local-secondary-index.html

Query and Scan

能不用 scan 就不用
畢竟這個操作就是去掃 table 裡的所有 item

primary key 和 local secondary index 只能在建立 table 時指定
一旦建立就不能改了
但是 global secondary index 就沒有這個限制

如果是用 partition + sork key 當 primary key
get 的時候要同時給 partition key 和 sort key
query 的時候可以只給 partition key 而 sort key 可給可不給（但是 partition key 一定要給）

無論是當 primary key、GSI 或 LSI
只要是 partition key 的 attribute 一律只能使用 = 來 query
該 attribute 沒有 rich query 的能力（就是 >, <, between, contains 那些條件）
sort key 才會有 rich query

Best Practices

http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/BestPractices.html

Choosing a Partition Key
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html

Querying DynamoDB by date
http://stackoverflow.com/questions/14836600/querying-dynamodb-by-date

Pick an item randomly
http://stackoverflow.com/questions/10666364/aws-dynamodb-pick-a-record-item-randomly

ref:
https://www.uplift.agency/blog/posts/2016/03/clearcare-dynamodb
https://medium.com/building-timehop/one-year-of-dynamodb-at-timehop-f761d9fe5fa1#.3g97b3lqy

Commands

DynamoDB is schema-less, so that you can only define keys you need for specifying primary key or local secondary index when creating table.

# 可以用 project name 作為 table name 的 prefix
# 之後可以隨時修改 read / write capacity units
$ aws dynamodb create-table \
--table-name CodeTengu_Preference \
--attribute-definitions AttributeName=name,AttributeType=S \
--key-schema AttributeName=name,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

$ aws dynamodb create-table \
--table-name CodeTengu_WeeklyIssue \
--attribute-definitions AttributeName=number,AttributeType=N AttributeName=publication,AttributeType=S AttributeName=publishedAt,AttributeType=N \
--key-schema AttributeName=number,KeyType=HASH \
--global-secondary-indexes IndexName=publication_publishedAt,KeySchema='[{AttributeName=publication,KeyType=HASH},{AttributeName=publishedAt,KeyType=RANGE}]',Projection='{ProjectionType=ALL}',ProvisionedThroughput='{ReadCapacityUnits=5,WriteCapacityUnits=5}' \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

$ aws dynamodb create-table \
--table-name CodeTengu_WeeklyPost \
--attribute-definitions AttributeName=issueNumber,AttributeType=N AttributeName=id,AttributeType=N  AttributeName=categoryCode,AttributeType=S \
--key-schema AttributeName=issueNumber,KeyType=HASH AttributeName=id,KeyType=RANGE \
--global-secondary-indexes IndexName=categoryCode_id,KeySchema='[{AttributeName=categoryCode,KeyType=HASH},{AttributeName=id,KeyType=RANGE}]',Projection='{ProjectionType=ALL}',ProvisionedThroughput='{ReadCapacityUnits=5,WriteCapacityUnits=5}' \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

ref:
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/create-table.html
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/update-table.html

$ aws dynamodb put-item \
--table-name CodeTengu_Preference \
--item file://fixtures/curated_api_config.json \
--return-consumed-capacity TOTAL

# fixtures/curated_api_config.json
{
  "name": { "S": "curated_api_config" },
  "apiKey": { "S": "xxx" }
}

ref:
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/put-item.html

$ aws dynamodb get-item \
--table-name CodeTengu_WeeklyIssue \
--key '{"number": {"N": "42"}}'

ref:
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/get-item.html

Usage

你應該用 AWS.DynamoDB.DocumentClient
而不是直接用 AWS.DynamoDB

const AWS = require("aws-sdk");

const dynamodb = new AWS.DynamoDB({
  apiVersion: "2012-08-10",
  region: "ap-northeast-1",
});
const dynamodbClient = new AWS.DynamoDB.DocumentClient({ service: dynamodb });

const params = {
  RequestItems: {
    CodeTengu_Preference: {
      Keys: [{ name: "xxx" }],
    },
  },
};

dynamodbClient.batchGet(params, (err, data) => {
  if (err) {
    console.log("fail");
    console.log(err);
  } else {
    console.log("success");
    console.log(data);
  }
});

ref:
http://aws.amazon.com/sdk-for-node-js/
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html

完整的程式碼放在 GitHub 上

https://github.com/CodeTengu/lambdabaku

argparse: Create a Command-line App with Python

2015-12-092026-03-17VintaPython

argparse is a Python standard library that makes it easy to write a CLI application. You should use this module instead of optparse which is deprecated since Python 2.7.

ref:
https://docs.python.org/3/library/argparse.html

Basic example

#!/usr/bin/env python

from __future__ import print_function

import argparse
import sys

import jokekappa

class JokeKappaCLI(object):

    def __init__(self):
        parser = argparse.ArgumentParser(
            prog='jokekappa',
            description='humor is a serious thing, you should take it seriously',
        )
        self.parser = parser
        self.parser.add_argument('-v', '--version', action='version', version=jokekappa.__version__)

        self.subparsers = parser.add_subparsers(title='sub commands')
        self.subparsers 
            .add_parser('one', help='print one joke randomly') 
            .set_defaults(func=self.tell_joke)
        self.subparsers 
            .add_parser('all', help='print all jokes') 
            .set_defaults(func=self.tell_jokes)
        self.subparsers 
            .add_parser('update', help='update jokes from sources') 
            .set_defaults(func=self.update_jokes)

    def parse_args(self):
        if len(sys.argv) == 1:
            namespace = self.parser.parse_args(['one', ])
        else:
            namespace = self.parser.parse_args()
        namespace.func()

    def tell_joke(self):
        joke = jokekappa.get_joke()
        print(joke['content'])

    def tell_jokes(self):
        for joke in jokekappa.get_jokes():
            print(joke['content'])

    def update_jokes(self):
        jokekappa.update_jokes()
        print('Done')

def main():
    JokeKappaCLI().parse_args()

if __name__ == '__main__':
    main()

ref:
https://github.com/CodeTengu/JokeKappa
https://stackoverflow.com/questions/5176691/argparse-how-to-specify-a-default-subcommand

Advanced example

In the following code, you're able to create a Python module called pangu and a command-line tool also called pangu, both share the same codebase.

in pangu.py

from __future__ import print_function

import argparse
import sys

__version__ = '3.3.0'
__all__ = ['spacing_text', 'PanguCLI']

def spacing_text(text):
    """
    You could find real code in https://github.com/vinta/pangu.py
    """
    return text.upper().strip()

class PanguCLI(object):

    def __init__(self):
        parser = argparse.ArgumentParser(
            prog='pangu',
            description='paranoid text spacing',
        )
        self.parser = parser
        self.parser.add_argument('-v', '--version', action='version', version=__version__)
        self.parser.add_argument('text', action='store', type=str)

    def parse(self):
        if not sys.stdin.isatty():
            print(spacing_text(sys.stdin.read()))
        elif len(sys.argv) > 1:
            namespace = self.parser.parse_args()
            print(spacing_text(namespace.text))
        else:
            self.parser.print_help()
        sys.exit(0)

if __name__ == '__main__':
    PanguCLI().parse()

in bin/pangu

#!/usr/bin/env python

from pangu import PanguCLI

if __name__ == '__main__':
    PanguCLI().parse()

in setup.py

from setuptools import setup

setup(
    name='pangu',
    py_modules=['pangu', ],
    scripts=['bin/pangu', ],
    ...
)

As a result, there're multiple usages:

$ pangu "abc"
$ python -m pangu "abc"
$ echo "abc" | pangu
$ echo "abc" | python -m pangu
ABC

ref:
https://github.com/vinta/pangu.py

Accept a list as option

parser.add_argument('-u', '--usernames', type=lambda x: x.split(','), dest='usernames', required=True)
# your_command -u vinta
# your_command -u vinta,saiday

Accept conditional argument

ref:
https://stackoverflow.com/questions/15459997/passing-integer-lists-to-python

awscli: Command-line Interface for Amazon Web Services

2015-09-232026-02-18VintaDevOps, Web Development

awscli is the official command-line interface for all Amazon Web Services (AWS).

ref:
https://github.com/aws/aws-cli

Configuration

$ pip install awscli

$ aws configure

ref:
https://docs.aws.amazon.com/cli/latest/index.html

S3

Download A Folder

$ aws s3 sync 
s3://files.vinta.ws/static/images/stickers/ 
.

ref:
https://docs.aws.amazon.com/cli/latest/reference/s3/sync.html
https://docs.aws.amazon.com/cli/latest/userguide/cli-services-s3-commands.html#using-s3-commands-managing-objects

Rename A Folder

$ aws s3 cp 
s3://files.vinta.ws/static/images/stickers_BACKUP/ 
s3://files.vinta.ws/static/images/stickers/ 
--recursive

ref:
https://docs.aws.amazon.com/cli/latest/reference/s3/cp.html

Make A Folder Public Read

$ aws s3 sync 
s3://files.vinta.ws/static/ 
s3://files.vinta.ws/static/ 
--grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers

Upload Files

# also make them public read
$ aws s3 cp 
. 
s3://files.vinta.ws/static/images/stickers/ 
--recursive 
--grants read=uri=http://acs.amazonaws.com/groups/global/AllUsers

$ aws s3 cp 
db.sqlite3 
s3://files.albedo.one/

$ aws s3 sync 
./ 
s3://files.albedo.one/ 
--recursive --exclude "*" --include "*.pickle"

Copy Files Between S3 Buckets

$ aws s3 sync s3://your_bucket_1/media s3://your_bucket_2/media 
--acl "public-read" 
--exclude "track_audio/*"

Remove Files

$ aws s3 rm s3://your_bucket_1/media/track_audio --recursive

ref:
https://docs.aws.amazon.com/cli/latest/reference/s3/rm.html

你都去哪裡看技術文章？

2015-09-012026-02-18VintaMisc

因為前陣子跟朋友們一起弄了一個技術週刊：CodeTengu Weekly 碼天狗週刊，每個禮拜在考慮要放哪些內容的時候，突然覺得：「你都去哪裡看技術文章？」或許也會是個有價值而且實用的主題，所以乾脆就來跟大家分享一下，我覺得不錯的每日資訊來源。

你可以訂閱的週報

五花八門

CodeTengu Weekly - 內容包含 Web 前端後端、Mobile app 開發、DevOps、機器學習
Hacker Newsletter - Hacker News 的每週精華版
码农周刊 - 开发者头条的每週精華版
湾区日报 - 講技術的部分其實不多，大部份說的是創業、管理、UX 之類的主題

程式語言

資料庫

DevOps

Machine Learning

你可以瀏覽的網站

如果要推薦值得一看的網站或網誌，說八年都說不完，而且現在大家也都不用 RSS reader 了（真的很可惜，明明就很方便），這裡就只提幾個「內容聚合網站」（news aggregator）。你可以在這些網站上 follow 特定的主題，例如 Python、Golang、Apache Cassandra、Docker 之類的，他們就會自動把相關的文章推送給你，比較特別的是，網站還會根據你的個人喜好和你在 Twitter 上關注的對象來調整推送給你的內容。

我最早用過這一類的服務是 Zite，但是直到它老是推薦「印度一條六公尺的巨蟒（Python）吞食了一個人類小孩」的新聞給我之後，我就把它刪掉了。雖然說 Zite 已經被收購，整合進 Flipboard 裡，但是我已經對它沒信心啦。

2015.09.06 更新：

你可以關注的人

以下列出的是許多喜歡在 Twitter 上分享技術文章而且推文頻率又比較高的開發者：

出沒於 Twitter

@vinta - 像這樣幽默又率性的 developer 在這個世道也是不多的，大家可以關注一下
@codetengu - 碼天狗的推特帳號
@c9s
@caterpillar
@dlackty
@Evan_Lin
@ferrari_tw
@fukuball
@gslin
@honglong0420
@hiroshiyui
@hsatac
@ihower
@ingramchen
@jaceju
@jserv
@kakashiliu
@keitheis
@lovecankill
@othree
@pahudnet
@ruanyf
@saiday
@siuying
@timdream
@tzangms
@uranusjr
@WanCW
@william_yeh
@yorkxin
@zmx

出沒於 Facebook

陳上進 - 是的，就是這篇文章的作者本人
CodeTengu 碼天狗 - 碼天狗的 Facebook Page
Ant Yi-Feng Tzeng
Chia-liang Kao
Ian Tsai
Mosky Liu
Will Huang