I Failed the Turing Test

AWS DynamoDB Notes

Posted on Author VintaPosted in Database, DevOps, Web Development

AWS DynamoDB is a fully managed key-value store (also document store) NoSQL database as a service provided by Amazon Web Services. Its pricing model is that you only pay for the throughput (read and write) you use instead of the storage usage and the running hours of database instances.

ref:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html
http://www.slideshare.net/AmazonWebServices/design-patterns-using-amazon-dynamodb

Glossary

DynamoDB is schema-less.

  • table: a table is a collection of items.
  • item: an item is a collection of attributes (key-value pairs).
  • attribute: attribute is similar to fields or columns in other databases.
  • primary key: one or two attributes that can uniquely identify every item in a table.
    • partition key (aka hash key): a simple primary key, composed of one attribute.
    • partition key and sort key (aka range key): a composite primary key, composed of two attributes.

ref:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/HowItWorks.CoreComponents.html

Global Secondary Index (GSI)

secondary index 指的是除了 primary key 之外的第二組 key
可以有很多組 secondary index
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/SecondaryIndexes.html

GSI 可以用在是 partition key 或 partition + sort key 的 table
GSI 跟 primary key 一樣可以 simple 或是 composite 的
GSI 可以隨時增減

如果你不需要 strong consistency 或個別 partition 的資料量大於 10GB
那就用 GSI

ref:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GSI.html
http://iamgarlic.blogspot.tw/2015/01/amazon-dynamodb-global-secondary-index.html

Local Secondary Index (LSI)

LSI 只能用在是 partition + sort key 的 table
LSI 必須用原本的 partition key 搭配其他 attribute 做為新的 partition + sort key（LSI 只會是 composite 的）
LSI 只能在建立 table 的時候定義

ref:
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/LSI.html
http://iamgarlic.blogspot.tw/2015/01/amazon-dynamodb-local-secondary-index.html

Query and Scan

能不用 scan 就不用
畢竟這個操作就是去掃 table 裡的所有 item

primary key 和 local secondary index 只能在建立 table 時指定
一旦建立就不能改了
但是 global secondary index 就沒有這個限制

如果是用 partition + sork key 當 primary key
get 的時候要同時給 partition key 和 sort key
query 的時候可以只給 partition key 而 sort key 可給可不給（但是 partition key 一定要給）

無論是當 primary key、GSI 或 LSI
只要是 partition key 的 attribute 一律只能使用 = 來 query
該 attribute 沒有 rich query 的能力（就是 >, <, between, contains 那些條件）
sort key 才會有 rich query

Best Practices
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/BestPractices.html

Choosing a Partition Key
http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/GuidelinesForTables.html

Querying DynamoDB by date
http://stackoverflow.com/questions/14836600/querying-dynamodb-by-date

Pick an item randomly
http://stackoverflow.com/questions/10666364/aws-dynamodb-pick-a-record-item-randomly

ref:
https://www.uplift.agency/blog/posts/2016/03/clearcare-dynamodb
https://medium.com/building-timehop/one-year-of-dynamodb-at-timehop-f761d9fe5fa1#.3g97b3lqy

Commands

DynamoDB is schema-less, so that you can only define keys you need for specifying primary key or local secondary index when creating table.

# 可以用 project name 作為 table name 的 prefix
# 之後可以隨時修改 read / write capacity units
$ aws dynamodb create-table \
--table-name CodeTengu_Preference \
--attribute-definitions AttributeName=name,AttributeType=S \
--key-schema AttributeName=name,KeyType=HASH \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

$ aws dynamodb create-table \
--table-name CodeTengu_WeeklyIssue \
--attribute-definitions AttributeName=number,AttributeType=N AttributeName=publication,AttributeType=S AttributeName=publishedAt,AttributeType=N \
--key-schema AttributeName=number,KeyType=HASH \
--global-secondary-indexes IndexName=publication_published_at,KeySchema='[{AttributeName=publication,KeyType=HASH},{AttributeName=publishedAt,KeyType=RANGE}]',Projection='{ProjectionType=ALL}',ProvisionedThroughput='{ReadCapacityUnits=5,WriteCapacityUnits=5}' \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

$ aws dynamodb create-table \
--table-name CodeTengu_WeeklyPost \
--attribute-definitions AttributeName=issueNumber,AttributeType=N AttributeName=id,AttributeType=N  AttributeName=categoryCode,AttributeType=S \
--key-schema AttributeName=issueNumber,KeyType=HASH AttributeName=id,KeyType=RANGE \
--global-secondary-indexes IndexName=categoryCode_id,KeySchema='[{AttributeName=categoryCode,KeyType=HASH},{AttributeName=id,KeyType=RANGE}]',Projection='{ProjectionType=ALL}',ProvisionedThroughput='{ReadCapacityUnits=5,WriteCapacityUnits=5}' \
--provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5

ref:
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/create-table.html
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/update-table.html

$ aws dynamodb put-item \
--table-name CodeTengu_Preference \
--item file://fixtures/curated_api_config.json \
--return-consumed-capacity TOTAL

# fixtures/curated_api_config.json
{
  "name": { "S": "curated_api_config" },
  "apiKey": { "S": "xxx" }
}

ref:
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/put-item.html

$ aws dynamodb get-item \
--table-name CodeTengu_WeeklyIssue \
--key '{"number": {"N": "42"}}'

ref:
http://docs.aws.amazon.com/cli/latest/reference/dynamodb/get-item.html

Usage

你應該用 AWS.DynamoDB.DocumentClient
而不是直接用 AWS.DynamoDB

const AWS = require('aws-sdk');

const dynamodb = new AWS.DynamoDB({ apiVersion: '2012-08-10', region: 'ap-northeast-1' });
const dynamodbClient = new AWS.DynamoDB.DocumentClient({ service: dynamodb });

const params = {
  RequestItems: {
    CodeTengu_Preference: {
      Keys: [
        { name: 'xxx' },
      ],
    },
  },
};

dynamodbClient.batchGet(params, (err, data) => {
  if (err) {
    console.log('fail');
    console.log(err);
  } else {
    console.log('success');
    console.log(data);
  }
});

ref:
http://aws.amazon.com/sdk-for-node-js/
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB.html
http://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/DocumentClient.html

完整的程式碼放在 GitHub 上
https://github.com/CodeTengu/lambdabaku