{"id":512,"date":"2018-02-14T20:25:34","date_gmt":"2018-02-14T12:25:34","guid":{"rendered":"https:\/\/vinta.ws\/code\/?p=512"},"modified":"2026-03-17T01:16:41","modified_gmt":"2026-03-16T17:16:41","slug":"mongodb-cookbook-queries-and-aggregations","status":"publish","type":"post","link":"https:\/\/vinta.ws\/code\/mongodb-cookbook-queries-and-aggregations.html","title":{"rendered":"MongoDB cookbook: Queries and Aggregations"},"content":{"rendered":"<p>Frequently accessed items are cached in memory, so that MongoDB can provide optimal response time.<\/p>\n<h2>MongoDB Shell in JavaScript<\/h2>\n<h3>Administration<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-javascript\">db.currentOp();\n\n\/\/ slow queries\ndb.currentOp({\n    \"active\": true,\n    \"secs_running\": {\"$gt\" : 3},\n    \"ns\": \/^test.\/\n});\n\n\/\/ queries not using any index\ndb.adminCommand({ \n    \"currentOp\": true,\n    \"op\": \"query\", \n    \"planSummary\": \"COLLSCAN\"\n});\n\n\/\/ operations with high numYields\ndb.adminCommand({ \n    \"currentOp\": true, \n    \"ns\": \/^test.\/, \n    \"numYields\": {\"$gte\": 100} \n}) \n\ndb.serverStatus().connections\n{\n    \"current\" : 269,\n    \"available\" : 838591,\n    \"totalCreated\" : 417342\n}<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/method\/db.currentOp\/\">https:\/\/docs.mongodb.com\/manual\/reference\/method\/db.currentOp\/<\/a><br \/>\n<a href=\"https:\/\/hackernoon.com\/mongodb-currentop-18fe2f9dbd68\">https:\/\/hackernoon.com\/mongodb-currentop-18fe2f9dbd68<\/a><br \/>\n<a href=\"http:\/\/www.mongoing.com\/archives\/6246\">http:\/\/www.mongoing.com\/archives\/6246<\/a><\/p>\n<h3>BSON Types<\/h3>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/bson-types\/\">https:\/\/docs.mongodb.com\/manual\/reference\/bson-types\/<\/a><\/p>\n<h3>Check If A Document Exists<\/h3>\n<p>It is significantly faster to use <code>find()<\/code> + <code>limit()<\/code> because <code>findOne()<\/code> will always read + return the document if it exists. <code>find()<\/code> just returns a cursor (or not) and only reads the data if you iterate through the cursor.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('message').find({_id: ObjectId(\"585836504b287b5022a3ae26\", delivered: false)}, {_id: 1}).limit(1)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/8389811\/how-to-query-mongodb-to-test-if-an-item-exists\">https:\/\/stackoverflow.com\/questions\/8389811\/how-to-query-mongodb-to-test-if-an-item-exists<\/a><br \/>\n<a href=\"https:\/\/blog.serverdensity.com\/checking-if-a-document-exists-mongodb-slow-findone-vs-find\/\">https:\/\/blog.serverdensity.com\/checking-if-a-document-exists-mongodb-slow-findone-vs-find\/<\/a><\/p>\n<h3>Find Documents<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('user').find({username: 'nanababy520'})\n\ndb.getCollection('message').find({_id: ObjectId(\"5a6383b8d93d7a3fadf75af3\")})\n\ndb.getCollection('message').find({_cls: 'Message'}).sort({posted_at: -1})\n\ndb.getCollection('message').find({sender: ObjectId(\"57aace67ac08e72acc3b265f\"), pricing: {$ne: 0}})\n\ndb.getCollection('message').find({\n    sender: ObjectId(\"5ac0f56038cfff013a123d85\"),\n    created_at: {\n        $gte: ISODate('2018-04-21 12:00:00Z'),\n        $lte: ISODate('2018-04-21 13:00:00Z')\n    }\n})\n.sort({created_at: -1})<\/code><\/pre>\n<h3>Find Documents With Regular Expression<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('user').find({'username': \/vicky\/})<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/query\/regex\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/query\/regex\/<\/a><\/p>\n<h3>Find Documents With An Array Field<\/h3>\n<ul>\n<li><code>$in: [...]<\/code> means &quot;intersection&quot; or &quot;any element in&quot;<\/li>\n<li><code>$all: [...]<\/code> means &quot;subset&quot; or &quot;contain&quot;<\/li>\n<li><code>$elemMatch: {...}<\/code> means &quot;any element match&quot;<\/li>\n<li><code>$not: {$elemMatch: {$nin: [...]}}<\/code> means &quot;subset&quot; or &quot;in&quot;<\/li>\n<\/ul>\n<p>The last one roughly means <code>not any([False, False, False, False])<\/code> where each <code>False<\/code> is indicating if the item is not in in <code>[...]<\/code>.<\/p>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/12223465\/mongodb-query-subset-of-an-array\">https:\/\/stackoverflow.com\/questions\/12223465\/mongodb-query-subset-of-an-array<\/a><\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('message').find({includes: ObjectId(\"5a4bb448af9c462c610d0cc7\")})\n\ndb.getCollection('user').find({gender: 'F', tags: 'promoted'})\ndb.getCollection('user').find({gender: 'F', 'tags.1': {$exists: true}})<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/query\/exists\/#exists-true\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/query\/exists\/#exists-true<\/a><\/p>\n<h3>Find Documents With An Array Field Of Embedded Documents<\/h3>\n<p>Usually, you could use <code>$elemMatch<\/code>.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">{'the_array_field': {'$elemMatch': {\n    'a_field_of_each_element': {'$lte': now},\n    'another_field_of_each_element': 123\n}}}<\/code><\/pre>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('message').find({\n    unlocks: {\n        $elemMatch: {\n            _cls: 'PointsUnlock',\n            user: ObjectId(\"57f662e727a79d07993faec5\")\n        }\n    }\n})\n\ndb.getCollection('feature.shop.product').find({\n    purchases: {\n        $elemMatch: {\n            _cls: 'Purchase'\n        }\n    }\n})\n\ndb.getCollection('feature.shop.product').find({\n    '_id': 'prod_CWlSTXBEU4mhEu',\n    'purchases': {'$not': {'$elemMatch': {\n        '_cls': 'DirectPurchase',\n        'user': ObjectId(\"58b61d9094ab56f912ba10a5\")\n    }}},\n})<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/query\/elemMatch\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/query\/elemMatch\/<\/a><\/p>\n<h3>Find Documents With Existence Of Fields Or Values<\/h3>\n<ul>\n<li><code>.find({&#039;field&#039;: {&#039;$exists&#039;: true}})<\/code>: the field exists<\/li>\n<li><code>.find({&#039;field&#039;: {&#039;$exists&#039;: false}})<\/code>: the field does not exist<\/li>\n<li><code>.find({&#039;field&#039;: {&#039;$type&#039;: 10}})<\/code>: the field exists with a <code>null<\/code> value<\/li>\n<li><code>.find({&#039;field&#039;: null})<\/code>: the field exists with a <code>null<\/code> value or the field does not exist<\/li>\n<li><code>.find({&#039;field&#039;: {&#039;$ne&#039;: null}})<\/code>: the field exists and the value is not <code>null<\/code><\/li>\n<li><code>.find({&#039;array_field&#039;: {&#039;$in&#039;: [null, []]}})<\/code><\/li>\n<\/ul>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.test.insert({'num': 1, 'check': 'value'})\ndb.test.insert({'num': 2, 'check': null})\ndb.test.insert({'num': 3})\n\ndb.test.find({});\n\ndb.test.find({'check': {'$exists': true}})\n\/\/ return 1 and 2\n\ndb.test.find({'check': {'$exists': false}})\n\/\/ return 3\n\ndb.test.find({'check': {'$type': 10}});\n\/\/ return 2\n\ndb.test.find({'check': null})\n\/\/ return 2 and 3\n\ndb.test.find({'check': {'$ne': null}});\n\/\/ return 1<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/4057196\/how-do-you-query-this-in-mongo-is-not-null\">https:\/\/stackoverflow.com\/questions\/4057196\/how-do-you-query-this-in-mongo-is-not-null<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/tutorial\/query-for-null-fields\/\">https:\/\/docs.mongodb.com\/manual\/tutorial\/query-for-null-fields\/<\/a><\/p>\n<h3>Find Documents Where An Array Field Does Not Contain A Certain Value<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('user').update({_id: ObjectId(\"579994ac61ff217f96a585d9\"), tags: {$ne: 'tag_to_add'}}, {$push: {tags: 'tag_to_add'}})\n\ndb.getCollection('user').update({_id: ObjectId(\"579994ac61ff217f96a585d9\"), tags: {$nin: ['tag_to_add']}}, {$push: {tags: 'tag_to_add'}})<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/16221599\/find-documents-with-arrays-not-containing-a-document-with-a-particular-field-val\">https:\/\/stackoverflow.com\/questions\/16221599\/find-documents-with-arrays-not-containing-a-document-with-a-particular-field-val<\/a><\/p>\n<h3>Find Documents Where An Array Field Is Not Empty<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('message').find({unlocks: {$exists: true}})<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/14789684\/find-mongodb-records-where-array-field-is-not-empty\">https:\/\/stackoverflow.com\/questions\/14789684\/find-mongodb-records-where-array-field-is-not-empty<\/a><\/p>\n<h3>Find Documents Where An Array Field's Size Is Greater Than 1<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('user.inbox').find({\n    'messages.0': {'$exists': true}\n})\n\ndb.getCollection('message').find({\n    '_cls': 'Message',\n    'unlocks.10': {'$exists': true}\n}).sort({'posted_at': -1})\n\ndb.getCollection('message').find({\n    '_cls': 'Message.ChatMessage',\n    'sender': ObjectId(\"582ee32a5b9c861c87dc297e\"),\n    'unlocks': {'$exists': true, '$not': {'$size': 0}}\n})<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/7811163\/query-for-documents-where-array-size-is-greater-than-1\/15224544\">https:\/\/stackoverflow.com\/questions\/7811163\/query-for-documents-where-array-size-is-greater-than-1\/15224544<\/a><\/p>\n<h3>Find Documents With Computed Values Using <code>$expr<\/code><\/h3>\n<p>For instance, compare 2 fields from a single document in a <code>find()<\/code> query.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('user').find({\n    $expr: {\n        $eq: [{$size: '$follows'}, {$size: '$blocks'}]\n    }\n})<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/thecodebarbarian.com\/a-nodejs-perspective-on-mongodb-36-lookup-expr\">https:\/\/thecodebarbarian.com\/a-nodejs-perspective-on-mongodb-36-lookup-expr<\/a><br \/>\n<a href=\"https:\/\/dzone.com\/articles\/expressive-query-language-in-mongodb-36-2\">https:\/\/dzone.com\/articles\/expressive-query-language-in-mongodb-36-2<\/a><\/p>\n<h3>Project A Subset Of An Array Field With <code>$filter<\/code><\/h3>\n<p>A sample document:<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">{\n    \"_id\" : \"message_unlock_pricing\",\n    \"seed\" : 42,\n    \"distributions\" : {\n        \"a\" : 0.5,\n        \"b\" : 0.5\n    },\n    \"whitelist\" : [ \n        {\n            \"_id\" : ObjectId(\"57dd071dd20fc40c0cbed6b7\"),\n            \"variation\" : \"a\"\n        }, \n        {\n            \"_id\" : ObjectId(\"5b1173a1487fbe2b2e9bba04\"),\n            \"variation\" : \"b\"\n        }, \n        {\n            \"_id\" : ObjectId(\"5a66d5c2af9c462c617ce552\"),\n            \"variation\" : \"b\"\n        }\n    ]\n}<\/code><\/pre>\n<pre class=\"line-numbers\"><code class=\"language-js\">var now = new Date();\n\ndb.getCollection('feature.ab.experiment').aggregate([\n    {'$project': {\n        '_id': 1,\n        'seed': 1,\n        'distributions': 1,\n        'whitelist': {\n            '$filter': {\n               'input': {'$ifNull': [\"$whitelist\", []]},\n               'as': \"user\",\n               'cond': {'$eq': ['$$user._id', ObjectId(\"5a66d5c2af9c462c617ce552\")]}\n            }\n         }\n    }},\n    {'$unwind': {\n        'path': '$whitelist',\n        'preserveNullAndEmptyArrays': true\n    }}\n])<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/42607221\/mongodb-aggregation-project-check-if-array-contains\">https:\/\/stackoverflow.com\/questions\/42607221\/mongodb-aggregation-project-check-if-array-contains<\/a><\/p>\n<h3>Insert Documents<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('feature.launch').insert({\n    'url': '\/\/example.com\/launchs\/5a06b88aaf9c462c6146ce12.jpg',\n    'user': {\n        'id': ObjectId(\"5a06b88aaf9c462c6146ce12\"),\n        'username': 'luke0804',\n        'tags': [\"gender:male\"]\n    }\n})\n\ndb.getCollection('feature.launch').insert({\n    'url': '\/\/example.com\/launchs\/57c16f5bb811055b66d8ef46.jpg',\n    'user': {\n        'id': ObjectId(\"57c16f5bb811055b66d8ef46\"),\n        'username': 'riva',\n        'tags': [\"gender:female\"]\n    }\n})<\/code><\/pre>\n<h3>Update Within A For Loop<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-js\">var oldTags = ['famous', 'newstar', 'featured', 'western', 'recommended', 'popular'];\noldTags.forEach(function(tag) {\n    db.getCollection('user').updateMany({tags: tag}, {$addToSet: {tags: 'badge:' + tag}});\n});<\/code><\/pre>\n<h3>Update With Conditions Of Field Values<\/h3>\n<p>You could update the value of the field to a specified value if the specified value is <strong>less than<\/strong> or <em>greater than<\/em> the current value of the field. The <code>$min<\/code> and <code>$max<\/code> operators can compare values of different types.<\/p>\n<p>Only set <code>posted_at<\/code> to current timestamp if its current value is None or absent.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">Post.objects.update_one(\n    {\n        '_id': bson.ObjectId(post_id),\n        'media.0': {'$exists': True},\n        'title': {'$ne': None},\n        'location': {'$ne': None},\n        'gender': {'$ne': None},\n        'pricing': {'$ne': None},\n    },\n    {\n        '$min': {'posted_at': utils.utcnow()},\n    },\n)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/min\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/min\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/max\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/max\/<\/a><\/p>\n<h3>Update An Array Field<\/h3>\n<p>Array update operators:<\/p>\n<ul>\n<li><code>$<\/code>: Acts as a placeholder to update the first element in an array for documents that matches the query condition.<\/li>\n<li><code>$[]<\/code>: Acts as a placeholder to update all elements in an array for documents that match the query condition.<\/li>\n<li><code>$[&lt;identifier&gt;]<\/code>: Acts as a placeholder to update elements in an array that match the <code>arrayFilters<\/code> condition.<\/li>\n<li><code>$addToSet<\/code>: Adds elements to an array only if they do not already exist in the set.<\/li>\n<li><code>$push<\/code>: Adds an item to an array.<\/li>\n<li><code>$pop<\/code>: Removes the first or last item of an array.<\/li>\n<li><code>$pull<\/code>: Removes all array elements that match a specified query.<\/li>\n<li><code>$pullAll<\/code>: Removes all matching values from an array.<\/li>\n<\/ul>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update-array\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update-array\/<\/a><br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/guide\/querying.html#atomic-updates\">http:\/\/docs.mongoengine.org\/guide\/querying.html#atomic-updates<\/a><br \/>\n<a href=\"http:\/\/thecodebarbarian.com\/a-nodejs-perspective-on-mongodb-36-array-filters.html\">http:\/\/thecodebarbarian.com\/a-nodejs-perspective-on-mongodb-36-array-filters.html<\/a><\/p>\n<p>Add an element in an array field.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">user_id = '582ee32a5b9c861c87dc297e'\ntag = 'my_tag'\n\nupdated = User.objects \n    .filter(id=user_id, tags__ne=tag) \n    .update_one(push__tags=tag)\n\nupdated = User.objects \n    .filter(id=user_id) \n    .update_one(add_to_set__schedules={\n        'tag': tag,\n         'nbf': datetime.datetime(2018, 6, 4, 0, 0),\n        'exp': datetime.datetime(2019, 5, 1, 0, 0),\n    })<\/code><\/pre>\n<p>Insert an element into an array at a certain position.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">slot = 2\nPost.objects.filter(id=post_id, media__id__ne=media_id).update_one(__raw__={\n    '$push': {\n        'media': {\n            '$each': [{'id': bson.ObjectId(media_id)}],\n            '$position': slot,\n        }\n    }\n})<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/position\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/position\/<\/a><br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/guide\/querying.html#querying-lists\">http:\/\/docs.mongoengine.org\/guide\/querying.html#querying-lists<\/a><\/p>\n<p>Remove elements in an array field. It is also worth noting that <code>update(pull__abc=xyz)<\/code> always returns <code>1<\/code>.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">user_id = '582ee32a5b9c861c87dc297e'\ntag = 'my_tag'\n\nupdated = User.objects \n    .filter(id=user_id) \n    .update_one(pull__tags=tag)\n\nupdated = User.objects \n    .filter(id=user_id) \n    .update_one(pull__schedules={'tag': tag})<\/code><\/pre>\n<p>Remove multiple embedded documents in an array field.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">import bson\n\nuser_id = '5a66d5c2af9c462c617ce552'\ntags = ['valid_tag_1', 'future_tag']\n\nupdated_result = User._get_collection().update_one(\n    {'_id': bson.ObjectId(user_id)},\n    {'$pull': {'schedules': {'tag': {'$in': tags}}}},\n)\nprint(updated_result.raw_result)\n# {'n': 1, 'nModified': 1, 'ok': 1.0, 'updatedExisting': True}<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/28102691\/pullall-while-removing-embedded-objects\">https:\/\/stackoverflow.com\/questions\/28102691\/pullall-while-removing-embedded-objects<\/a><\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('feature.feeds').updateMany(\n    {\n        'aliases': {'$exists': true},\n        'exp': {'$gte': ISODate('2019-03-21T00:00:00.000+08:00')},\n        'items': {'$elemMatch': {'username': 'engine'}},\n    },\n    {\n        '$pull': {\n            'items': {'username': 'engine'},\n        }\n    }\n);<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/pull\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/pull\/<\/a><\/p>\n<p>You could also use <code>add_to_set<\/code> to add an item to an array only if it is not in the list, which always returns <code>1<\/code> if <code>filter()<\/code> matches any document. However, you are able to set <code>full_result=True<\/code> to get detail updated result.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">update_result = User.objects.filter(id=user_id).update_one(\n    add_to_set__tags=tag,\n    full_result=True,\n)\n# {'n': 1, 'nModified': 1, 'ok': 1.0, 'updatedExisting': True}<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/guide\/querying.html#atomic-updates\">http:\/\/docs.mongoengine.org\/guide\/querying.html#atomic-updates<\/a><\/p>\n<p>Update a multi-level nested array field. Yes, <code>arrayFilters<\/code> supports it.<\/p>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/positional-filtered\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/positional-filtered\/<\/a><br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/23577123\/updating-a-nested-array-with-mongodb\">https:\/\/stackoverflow.com\/questions\/23577123\/updating-a-nested-array-with-mongodb<\/a><\/p>\n<p>Update an embedding document in an array field.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">MessagePackProduct.objects \n    .filter(id='prod_CR1u34BIpDbHeo', skus__id='sku_CR23rZOTLhYprP') \n    .update(__raw__={\n        '$set': {'skus.$': {'id': 'sku_CR23rZOTLhYprP', 'test': 'test'}}\n    })<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/9200399\/replacing-embedded-document-in-array-in-mongodb\">https:\/\/stackoverflow.com\/questions\/9200399\/replacing-embedded-document-in-array-in-mongodb<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/method\/db.collection.update\/#db.collection.update\">https:\/\/docs.mongodb.com\/manual\/reference\/method\/db.collection.update\/#db.collection.update<\/a><\/p>\n<p>Update specific embedded documents with <code>arrayFilters<\/code> in an array field.<\/p>\n<p>User data:<\/p>\n<pre class=\"line-numbers\"><code class=\"language-json\">{\n    \"_id\" : ObjectId(\"5a66d5c2af9c462c617ce552\"),\n    \"username\" : \"gibuloto\",\n    \"tags\" : [\n        \"beta\",\n        \"future_tag\",\n        \"expired_tag\"\n    ],\n    \"schedules\" : [\n        {\n            \"tag\" : \"valid_tag\",\n            \"nbf\" : ISODate(\"2018-05-01T16:00:00.000Z\"),\n            \"exp\" : ISODate(\"2020-06-04T16:00:00.000Z\")\n        },\n        {\n            \"tag\" : \"future_tag\",\n            \"nbf\" : ISODate(\"2020-01-28T16:00:00.000Z\"),\n            \"exp\" : ISODate(\"2020-12-14T16:00:00.000Z\")\n        },\n        {\n            \"tag\" : \"expired_tag\",\n            \"nbf\" : ISODate(\"2016-02-12T16:00:00.000Z\"),\n            \"exp\" : ISODate(\"2016-04-21T16:00:00.000Z\")\n        }\n    ],\n}<\/code><\/pre>\n<p>It is worth noting that <code>&lt;identifier&gt;<\/code> in <code>$arrayFilters<\/code> can only contain lowercase alphanumeric characters.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">import bson\n\nuser_id = '5a66d5c2af9c462c617ce552'\ntags = ['from_past_to_future']\n\nupdated_result = User._get_collection().update_one(\n    {'_id': bson.ObjectId(user_id)},\n    {\n        '$addToSet': {'tags': {'$each': tags}},\n        '$unset': {'schedules.$[schedule].nbf': True},\n    },\n    array_filters=[{'schedule.tag': {'$in': [tag for tag in tags]}}],\n)\nprint(updated_result.raw_result)\n# {'n': 1, 'nModified': 1, 'ok': 1.0, 'updatedExisting': True}<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/master\/reference\/operator\/update\/positional-filtered\/\">https:\/\/docs.mongodb.com\/master\/reference\/operator\/update\/positional-filtered\/<\/a><\/p>\n<h3>Update An Array Field With <code>arrayFilters<\/code><\/h3>\n<p>You should use <code>arrayFilters<\/code> as much as possible.<\/p>\n<p>The syntax of <code>arrayFilters<\/code> would be:<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.collection.update(\n   {&lt;query selector&gt;},\n   {&lt;update operator&gt;: {'array.$[&lt;identifier&gt;].field': value}},\n   {arrayFilters: [{&lt;identifier&gt;: &lt;condition&gt;}}]}\n)<\/code><\/pre>\n<pre class=\"line-numbers\"><code class=\"language-py\">Inbox._get_collection().update_many(\n    {'messages.id': message_id},\n    {'$set': {'messages.$[message].tags': tags}},\n    array_filters=[\n        {'message.id': message_id},\n    ],\n)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/positional-filtered\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/positional-filtered\/<\/a><\/p>\n<p>Insert an element into an array field at a certain position.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('feature.forums.post').update(\n   { _id: ObjectId(\"5b3c6a9c8433b15569cae54e\") },\n   {\n     $push: {\n        media: {\n           $each: [{\n                \"mimetype\" : \"image\/jpeg\",\n                \"url\" : \"https:\/\/example.com\/posts\/5adb795b47d057338abe8910.jpg\",\n                \"presets\" : {}\n            }],\n           $position: 1\n        }\n     }\n   }\n)<\/code><\/pre>\n<p>Or use explicit array index <code>$set<\/code>.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">media_id = 'xxx'\nmedia_slot = 0\n\nPost.objects \n    .filter(id=post_id, **{f'media__{media_slot}__id__ne': media_id}) \n    .update_one(__raw__={'$set': {f'media.{media_slot}': {'id': media_id}}})<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/position\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/position\/<\/a><\/p>\n<p>Set an array field to empty.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('message').update(\n    {'tags': 'pack:joycelai-1'},\n    {'$set': {'unlocks': []}},\n    {'multi': true}\n)\n\ndb.getCollection('feature.shop.product').update(\n    {},\n    {'$set': {'purchases': []}},\n    {'multi': true}\n)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/method\/db.collection.update\/\">https:\/\/docs.mongodb.com\/manual\/reference\/method\/db.collection.update\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/set\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/set\/<\/a><\/p>\n<p>Remove elements from an array field.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">var userId = ObjectId(\"57985b784af4124063f090d3\");\n\ndb.getCollection('user').update(\n    {'follows.user': userId},\n    {'$pull': {'follows': {'user': userId}}},\n    {\n        'multi': true,\n    }\n);\n\ndb.getCollection('message').update(\n    {'_id': {'$in': [\n        ObjectId('5aca1ffc4271ab1624787ec4'),\n        ObjectId('5aca31ab93ef2936291c3dd4'),\n        ObjectId('5aca33d9b5eaef04943c0d0b'),\n        ObjectId('5aca34e7a48c543b07fb0a0f'),\n        ObjectId('5aca272d93ef296edc1c3dee'),\n        ObjectId('5aca342aa48c54306dfb0a21'),\n        ObjectId('5aca20756bd01023a8cb02e9')\n    ]}},\n    {'$pull': {'tags': 'pack:prod_D75YlDMzcCiAw3'}},\n    {'multi': true}\n);<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/pull\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/pull\/<\/a><\/p>\n<h3>Update A Dictionary Field<\/h3>\n<p>Set a key\/value in a dictionary field.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">tutorial.data = {\n    \"price_per_message\": 1200,\n    \"inbox\": []\n}\n\nnew_inbox = [\n    {\n        \"id\": \"5af118c598eacb528e8fb8f9\",\n        \"sender\": \"5a13239eaf9c462c611510fc\"\n    },\n    {\n        \"id\": \"5af1117298eacb212a8fb8e9\",\n        \"sender\": \"5a99554be9a21d5ff38b8ca5\"\n    }\n]\ntutorial.update(set__data__inbox=new_inbox)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/21158028\/updating-a-dictfield-in-mongoengine\">https:\/\/stackoverflow.com\/questions\/21158028\/updating-a-dictfield-in-mongoengine<\/a><\/p>\n<h3>Upsert: Update Or Create<\/h3>\n<p>You must use <code>upsert=true<\/code> with uniquely indexed fields. If you don't need the modified document, you should just use <code>update_one(field1=123, field2=456, upsert=True)<\/code>.<\/p>\n<p>Additionally, remember that <code>modify()<\/code> always reloads the whole object even the original one only loads specific fields with <code>only()<\/code>. Try to avoid using <code>document.DB_QUERY_METHOD()<\/code>, and using <code>User.objects.filter().only().modify()<\/code> or <code>User.objects.filter().update()<\/code> when it is possible.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">tag_schedule = TagSchedule.objects \n    .filter(user=user_id, tag='vip') \n    .modify(\n        started_at=started_at,\n        ended_at=ended_at,\n        upsert=True\n    )\n\nuser = User.objects \n    .filter(id=user.id, tutorials__buy_diamonds__version=None) \n    .modify(set__tutorials__buy_diamonds__version='v1')\n\nupdated = User.objects \n    .filter(user=user_id, tag=tag) \n    .update_one(\n        push__followers=new_follower,\n    )<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/method\/db.collection.update\/#update-with-unique-indexes\">https:\/\/docs.mongodb.com\/manual\/reference\/method\/db.collection.update\/#update-with-unique-indexes<\/a><br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/apireference.html#mongoengine.queryset.QuerySet.modify\">http:\/\/docs.mongoengine.org\/apireference.html#mongoengine.queryset.QuerySet.modify<\/a><br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/apireference.html#mongoengine.queryset.QuerySet.update_one\">http:\/\/docs.mongoengine.org\/apireference.html#mongoengine.queryset.QuerySet.update_one<\/a><\/p>\n<h3>Rename A Field<\/h3>\n<p>Simply rename a field with <code>$rename<\/code>.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('user').updateMany(\n    {\n        'phone_no': {'$exists': true},\n        'social.phone-number.uid': {'$exists': false},\n    },\n    {'$rename': {\n        'phone_no': 'social.phone-number.uid',\n    }}\n);<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/rename\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/update\/rename\/<\/a><\/p>\n<p>Do some extra data converting and rename the field manually.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.getCollection('user').aggregate([\n    {'$match': {\n        'twitter.id': {'$exists': true},\n        'social.twitter.uid': {'$exists': false},\n    }},\n    {'$project': {\n        'twitter_id': '$twitter.id',\n        'twitter_id_str': {'$toString': '$twitter.id'},\n    }},\n]).forEach(function (document) {\n    printjson({\n        'id': document._id,\n    });\n\n    db.getCollection('user').updateOne(\n      {\n          'twitter.id': document.twitter_id,\n          'social.twitter.uid': {'$exists': false},\n      },\n      {\n          '$unset': {'twitter.id': true},\n          '$set': {'social.twitter.uid': document.twitter_id_str}\n      }\n    )\n})<\/code><\/pre>\n<h3>Insert\/Replace Large Amount Of Documents<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-js\">const operations = contracts.map((contract) =&gt; {\n    \/\/ TODO: should create a new contract if there is any change of the contract?\n    \/\/ use MongoDB transaction to change the new one and old one\n    return {\n        'replaceOne': {\n            'filter': {'settlement_datetime': currentSettlementMonth.toDate(), 'user': contract.user},\n            'replacement': contract,\n            'upsert': true,\n        },\n    };\n});\n\ndb.collection('user.contract').bulkWrite(\n    operations,\n    {ordered: true},\n    (bulkError, result) =&gt; {\n        if (bulkError) {\n            return next(bulkError, null);\n        }\n\n        logger.info('Finished importing all contracts');\n        return next(null, result);\n    },\n);<\/code><\/pre>\n<h3>Update Large Numbers Of Documents<\/h3>\n<p>Use <code>Bulk.find.arrayFilters()<\/code> and <code>Bulk.find.update()<\/code> together.<\/p>\n<p>In Python:<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">import datetime\n\nexpiration_time = datetime.datetime.utcnow() - datetime.timedelta(hours=48)\n\nbulk = Outbox._get_collection().initialize_unordered_bulk_op()\n\nfor outbox in Outbox.objects.only('id').filter(messages__posted_at__lt=expiration_time):\n    bulk.find({'_id': outbox.id}).update_one({\n        '$pull': {'messages': {\n            'posted_at': {'$lt': expiration_time},\n        }},\n    })\n\ntry:\n    results = bulk.execute()\nexcept pymongo.errors.InvalidOperation as err:\n    if str(err) != 'No operations to execute':\n        raise err<\/code><\/pre>\n<p>In JavaScript:<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">const operations = docs.map((doc) =&gt; {\n    logger.debug(doc, 'Revenue');\n\n    const operation = {\n        'updateOne': {\n            'filter': {\n                '_id': doc._id,\n            },\n            'update': {\n                '$set': {\n                    'tags': doc.contract.tags,\n                },\n            },\n        },\n    };\n    return operation;\n});\n\ndb.collection('user.revenue').bulkWrite(\n    operations,\n    {ordered: false},\n    (bulkError, bulkResult) =&gt; {\n        if (bulkError) {\n            return next(bulkError, null);\n        }\n\n        logger.info(bulkResult, 'Saved tags');\n        return next(null, true);\n    },\n);\n});<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/method\/Bulk\/\">https:\/\/docs.mongodb.com\/manual\/reference\/method\/Bulk\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/method\/Bulk.find.arrayFilters\/\">https:\/\/docs.mongodb.com\/manual\/reference\/method\/Bulk.find.arrayFilters\/<\/a><\/p>\n<p>Of course, you could also update the same document with multiple operations. However, it does not make sense.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">from pymongo import UpdateOne\nimport bson\n\ndef _operations():\n    if title = payload.get('title'):\n        yield UpdateOne({'_id': bson.ObjectId(post_id)}, {'$set': {'title': title}})\n\n    if location = payload.get('location'):\n        yield UpdateOne({'_id': bson.ObjectId(post_id)}, {'$set': {'location': location}})      \n\n    if pricing = payload.get('pricing'):\n        yield UpdateOne({'_id': bson.ObjectId(post_id)}, {'$set': {'pricing': pricing}})\n\n    if description = payload.get('description'):\n        yield UpdateOne({'_id': bson.ObjectId(post_id)}, {'$set': {'description': description}})\n\n    UpdateOne(\n        {\n            '_id': bson.ObjectId(post_id),\n            'media.0': {'$exists': True},\n            'title': {'$ne': None},\n            'location': {'$ne': None},\n            'pricing': {'$ne': None},\n            'posted_at': {'$eq': None},\n        },\n        {'$set': {'posted_at': utils.utcnow()}},\n    )\n\noperations = list(_operations())\nresult = Post._get_collection().bulk_write(operations, ordered=True)\nprint(result.bulk_api_result)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/api.mongodb.com\/python\/current\/examples\/bulk.html\">https:\/\/api.mongodb.com\/python\/current\/examples\/bulk.html<\/a><\/p>\n<p>Remove items from an array field of documents.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">var userId = ObjectId(\"57a42a779f22bb6bcc434520\");\n\ndb.getCollection('user').update(\n    {'follows.user': userId},\n    {'$pull': {'follows': {'user': userId}}},\n    {'multi': true}\n)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/33594397\/how-to-update-a-large-number-of-documents-in-mongodb-most-effeciently\">https:\/\/stackoverflow.com\/questions\/33594397\/how-to-update-a-large-number-of-documents-in-mongodb-most-effeciently<\/a><\/p>\n<h3>Remove Large Numbers Of Documents<\/h3>\n<p>in <code>mongo<\/code> shell:<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">var bulk = db.getCollection('feature.journal.v2').initializeUnorderedBulkOp()\nbulk.find({}).remove()\nbulk.execute()\n\n\/\/ or\n\nvar bulk = db.getCollection('feature.journal.v2').initializeUnorderedBulkOp()\nbulk.find({event: 'quest.rewarded'}).remove()\nbulk.find({event: 'message.sent'}).remove()\nbulk.execute()<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/method\/Bulk.find.remove\/#bulk-find-remove\">https:\/\/docs.mongodb.com\/manual\/reference\/method\/Bulk.find.remove\/#bulk-find-remove<\/a><\/p>\n<h2>MongoEngine In Python<\/h2>\n<p>ref:<br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/guide\/index.html\">http:\/\/docs.mongoengine.org\/guide\/index.html<\/a><br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/apireference.html\">http:\/\/docs.mongoengine.org\/apireference.html<\/a><\/p>\n<h3>Define Collections<\/h3>\n<p>It seems every collection in MongoEngine must have a <code>id<\/code> field.<\/p>\n<p>ref:<br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/guide\/defining-documents.html\">http:\/\/docs.mongoengine.org\/guide\/defining-documents.html<\/a><\/p>\n<h3>Define A Field With Default <code>EmbeddedDocument<\/code><\/h3>\n<p>The behavior of setting an <code>EmbeddedDocument<\/code> class as default works differently with and without <code>only()<\/code>.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">class User(ab.models.ABTestingMixin, db.Document):\n    class UserSettings(db.EmbeddedDocument):\n        reply_price = db.IntField(min_value=0, default=500, required=True)\n        preferences = db.ListField(db.StringField())\n\n    email = db.EmailField(max_length=255)\n    created_at = db.DateTimeField(default=utils.now)\n    last_active = db.DateTimeField(default=utils.now)\n    settings = db.EmbeddedDocumentField(UserSettings, default=UserSettings)<\/code><\/pre>\n<p>If the user does not have <code>settings<\/code> field in DB, here is the difference.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">user = User.objects.get(username='gibuloto')\nisinstance(user.settings, User.UserSettings) == True\n\nuser = User.objects.only('settings').get(username='gibuloto')\n(user.settings is None) == True\n\nuser = User.objects.exclude('settings').get(username='gibuloto')\nisinstance(user.settings, User.UserSettings) == True<\/code><\/pre>\n<h3>Filter With Raw Queries<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">post = Post.objects \n    .no_dereference().only('posted_at') \n    .filter(__raw__={\n        '_id': bson.ObjectId(post_id),\n        'media.0': {'$exists': True},\n        'title': {'$ne': None},\n        'location': {'$ne': None},\n        'gender': {'$ne': None},\n        'pricing': {'$ne': None},\n    }) \n    .modify(__raw__={'$min': {'posted_at': utils.utcnow()}}, new=True)\n\nprint(post.posted_at)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/guide\/querying.html#raw-queries\">http:\/\/docs.mongoengine.org\/guide\/querying.html#raw-queries<\/a><\/p>\n<h3>Check If A Document Exists<\/h3>\n<p>Use <code>.exists()<\/code>.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">import datetime\n\nnow = datetime.datetime.now(datetime.timezone.utc)\nif TagSchedule.objects.filter(user=user_id, tag=tag, started_at__gt=now).exists():\n    return 'exists'<\/code><\/pre>\n<p>You have to use <code>__raw__<\/code> if the field you want to query is a <code>db.ListField(GenericEmbeddedDocumentField(XXX)<\/code> field.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">if MessagePackProduct.objects.filter(id=message_pack_id, __raw__={'purchases.user': g.user.id}).exists():\n    return 'exists'<\/code><\/pre>\n<h3>Upsert: Get Or Create<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">buy_diamonds = BuyDiamonds.objects.filter(user_id=user.id).upsert_one()<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/apireference.html#mongoengine.queryset.QuerySet.upsert_one\">http:\/\/docs.mongoengine.org\/apireference.html#mongoengine.queryset.QuerySet.upsert_one<\/a><\/p>\n<h3>Store Files On GridFS<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\"># models.py\nclass User(db.Document):\n    username = db.StringField()\n    image = db.ImageField(collection_name='user.images')<\/code><\/pre>\n<pre class=\"line-numbers\"><code class=\"language-py\"># tasks.py\nimport bson\nimport gridfs\nimport mongoengine\n\n@celery.shared_task(bind=True, ignore_result=True)\ndef gridfs_save(task, user_id, format='JPEG', raw_data: bytes=None, **kwargs):\n    image_id = None\n\n    if raw_data is None:\n        user = User.objects.only('image').get(id=user_id)\n        if user.image.grid_id:\n            image_id, raw_data = user.image.grid_id, user.image.read()\n\n    if not raw_data:\n        return\n\n    gf = gridfs.GridFS(mongoengine.connection.get_db(), User.image.collection_name)\n\n    with io.BytesIO(raw_data) as raw_image:\n        with Image.open(raw_image) as image:\n            image = image.convert('RGB')\n            with io.BytesIO() as buffer:\n                image.save(buffer, format=format, quality=80, **kwargs)\n                buffer.seek(0)\n                grid_id = gf.put(buffer, format=format, width=image.width, height=image.height, thumbnail_id=None)\n\n    # NOTE: If function was passed with raw_data, only override if ID is the same as the read\n    query = mongoengine.Q(id=user_id)\n    if image_id:\n        query = query &amp; mongoengine.Q(image=image_id)\n\n    user = User.objects.only('image').filter(query).modify(\n        __raw__={'$set': {'image': grid_id}},\n        new=False,\n    )\n\n    def cleanup():\n        # Delete the old image\n        if user and user.image:\n            yield user.image.grid_id\n\n        # The user image was already changed before the scheduled optimization took place\n        # Drop the optimized image\n        if user is None and image_id:\n            yield image_id\n\n    gridfs_delete.apply_async(kwargs=dict(\n        collection=User.image.collection_name,\n        grid_ids=list(cleanup()),\n    ))\n\n@celery.shared_task(bind=True, ignore_result=True)\ndef gridfs_delete(task, collection, grid_ids):\n    gf = gridfs.GridFS(mongoengine.connection.get_db(), collection)\n    for grid_id in grid_ids:\n        gf.delete(bson.ObjectId(grid_id))<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"http:\/\/docs.mongoengine.org\/guide\/gridfs.html\">http:\/\/docs.mongoengine.org\/guide\/gridfs.html<\/a><\/p>\n<h2>Store Datetime<\/h2>\n<p>MongoDB stores datetimes in UTC.<\/p>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/method\/Date\/\">https:\/\/docs.mongodb.com\/manual\/reference\/method\/Date\/<\/a><\/p>\n<h2>2-phase Commit<\/h2>\n<p>The easiest way to think about 2-phase commit is idempotency, i.e., if you run an update many times, the results would &quot;be the same&quot;: initial -&gt; pending -&gt; applied -&gt; done.<\/p>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/tutorial\/perform-two-phase-commits\/\">https:\/\/docs.mongodb.com\/manual\/tutorial\/perform-two-phase-commits\/<\/a><\/p>\n<h2>Aggregation Pipeline<\/h2>\n<ul>\n<li><code>$match<\/code>: Filters documents.<\/li>\n<li><code>$project<\/code>: Modifies document fields.<\/li>\n<li><code>$addFields<\/code>: Adds or overrides document fields.<\/li>\n<li><code>$group<\/code>: Groups documents by fields.<\/li>\n<li><code>$lookup<\/code>: Joins another collection.<\/li>\n<li><code>$replaceRoot<\/code>: Promotes an embedded document field to the top level and replaces all other fields.<\/li>\n<li><code>$unwind<\/code>: Expands an array field into multiple documents along with original documents.<\/li>\n<li><code>$facet<\/code>: Processes multiple pipelines within one stage and output to different fields.<\/li>\n<\/ul>\n<p>There are special system variables, for instance, <code>$$ROOT<\/code>, <code>$$REMOVE<\/code>, <code>$$PRUNE<\/code>, which you could use in some stages of the aggregation pipeline.<\/p>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/aggregation-variables\/#system-variables\">https:\/\/docs.mongodb.com\/manual\/reference\/aggregation-variables\/#system-variables<\/a><\/p>\n<h3>Return Date As Unix Timestamp<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">import datetime\n\ndef stages():\n    yield {'$project': {\n        'createdAt': {'$floor': {'$divide': [{'$subtract': ['$$created', datetime.datetime.utcfromtimestamp(0)]}, 1000]}},\n    }}\n\ntry:\n    docs = MessagePackProduct.objects.aggregate(*stages())\nexcept StopIteration:\n    docs = []\nelse:\n    for doc in docs:\n        print(doc)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/39274311\/convert-iso-date-to-timestamp-in-mongo-query\">https:\/\/stackoverflow.com\/questions\/39274311\/convert-iso-date-to-timestamp-in-mongo-query<\/a><\/p>\n<h3>Match Multiple Conditions Which Store In An Array Fields<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-json\">db.getCollection('feature.promotions').insert({\n    \"name\": \"TEST\",\n    \"nbf\": ISODate(\"2018-05-31 16:00:00.000Z\"),\n    \"exp\": ISODate(\"2018-06-30 15:59:00.001Z\"),\n    \"positions\": {\n        \"discover\": {\n            \"urls\": [\n                \"https:\/\/example.com\/events\/2018\/jun\/event1\/banner.html\"\n            ]\n        }\n    },\n    \"requirements\" : [\n        {\n            \/\/ users who like women and their app version is greater than v2.21\n            \"preferences\" : [\n                \"gender:female\"\n            ],\n            \"version_major_min\": 2.0,\n            \"version_minor_min\": 21.0\n        },\n        {\n            \/\/ female CPs\n            \"tags\" : [\n                \"stats\",\n                \"gender:female\"\n            ]\n        }\n    ]\n});<\/code><\/pre>\n<pre class=\"line-numbers\"><code class=\"language-py\">import werkzeug\n\nuser_agent = werkzeug.UserAgent('hello-world\/2.25.1 (iPhone; iOS 11.4.1; Scale\/2.00; com.example.app; zh-tw)')\nuser_preferences = ['gender:female', 'gender:male']\nuser_tags = ['beta', 'vip']\nuser_platforms = [user_agent.platform]\n\ndef stages():\n    now = utils.utcnow()\n\n    yield {'$match': {\n        '$and': [\n            {'nbf': {'$lte': now}},\n            {'exp': {'$gt': now}},\n            {'requirements': {'$elemMatch': {\n                'preferences': {'$not': {'$elemMatch': {'$nin': user_preferences}}},\n                'tags': {'$not': {'$elemMatch': {'$nin': user_tags}}},\n                'platforms': {'$not': {'$elemMatch': {'$nin': user_platforms}}},\n                '$or': [\n                    {'$and': [\n                        {'version_major_min': {'$lte': user_agent.version.major}},\n                        {'version_minor_min': {'$lte': user_agent.version.minor}},\n                    ]},\n                    {'$and': [\n                        {'version_minor_min': {'$exists': False}},\n                        {'version_minor_min': {'$exists': False}},\n                    ]},\n                ],\n            }}},\n        ],\n    }}\n    yield {'$project': {\n        'name': True,\n        'nbf': True,\n        'exp': True,\n        'positions': {'$objectToArray': '$positions'},\n    }}\n    yield {'$unwind': '$positions'}\n    yield {'$sort': {\n        'exp': 1,\n    }}\n    yield {'$project': {\n        '_id': False,\n        'name': True,\n        'position': '$positions.k',\n        'url': {'$arrayElemAt': ['$positions.v.urls', 0]},\n        'startedAt': {'$floor': {'$divide': [{'$subtract': ['$nbf', constants.UNIX_EPOCH]}, 1000]}},\n        'endedAt': {'$floor': {'$divide': [{'$subtract': ['$exp', constants.UNIX_EPOCH]}, 1000]}},\n    }}\n    yield {'$group': {\n        '_id': '$position',\n        'items': {'$push': '$$ROOT'},\n    }}\n\ntry:\n    docs = Promotion.objects.aggregate(*stages())\nexcept StopIteration:\n    docs = []\nelse:\n    docs = list(docs)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/query\/in\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/query\/in\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/query\/nin\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/query\/nin\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/setIsSubset\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/setIsSubset\/<\/a><\/p>\n<h3>Do Distinct With <code>$group<\/code><\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    yield {'$match': {\n        'tags': 'some_tag',\n    }}\n    yield {'$unwind': '$unlocks'}\n    yield {'$replaceRoot': {'newRoot': '$unlocks'}}\n    yield {'$match': {\n        '_cls': 'MessagePackUnlock',\n    }}\n    yield {'$group': {\n        '_id': '$user',\n        'timestamp': {'$first': '$timestamp'},\n    }}\n\nfor unlock in MessagePackMessage.objects.aggregate(*stages()):\n    tasks.offline_purchase_pack.apply(kwargs=dict(\n        user_id=unlock['_id'],\n        message_pack_id=message_pack.id,\n        timestamp=unlock['timestamp'],\n    ))<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/group\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/group\/<\/a><\/p>\n<h3>Slice Items In Each <code>$group<\/code><\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">import random\n\ndef stages():\n    yield {'$match': {'tags': {'$regex': '^badge:'}}}\n    yield {'$unwind': {'path': '$tags', 'includeArrayIndex': 'index'}}\n    yield {'$match': {'tags': {'$regex': '^badge:'}}}\n    yield {'$project': {'_id': True, 'tag': '$tags', 'index': {'$mod': ['$index', random.random()]}}}\n    yield {'$sort': {'index': 1}}\n    yield {'$group': {'_id': '$tag', 'users': {'$addToSet': '$_id'}}}\n    yield {'$project': {'_id': True, 'users': {'$slice': ['$users', 1000]}}}\n\ndocs = User.objects.aggregate(*stages())\nfor doc in docs:\n    badge, user_ids = doc['_id'], doc['users']<\/code><\/pre>\n<h3>Collect Items With <code>$group<\/code> And <code>$addToSet<\/code><\/h3>\n<p>User data:<\/p>\n<pre class=\"line-numbers\"><code class=\"language-json\">{\n    \"_id\" : ObjectId(\"5a66d5c2af9c462c617ce552\"),\n    \"username\" : \"gibuloto\",\n    \"tags\" : [ \n        \"beta\"\n    ],\n    \"schedules\" : [ \n        {\n            \"tag\" : \"stats\",\n            \"nbf\" : ISODate(\"2018-02-01T16:00:00.000Z\"),\n            \"exp\" : ISODate(\"2018-08-12T16:00:00.000Z\")\n        }, \n        {\n            \"tag\" : \"vip\",\n            \"nbf\" : ISODate(\"2018-05-13T16:00:00.000Z\"),\n            \"exp\" : ISODate(\"2018-05-20T16:00:00.000Z\")\n        }\n    ]\n}<\/code><\/pre>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    now = utils.utcnow()\n\n    yield {'$match': {\n        'schedules': {'$elemMatch': {\n            'nbf': {'$lte': now},\n            'exp': {'$gte': now}\n        }}\n    }}\n    yield {'$unwind': '$schedules'}\n    yield {'$match': {\n        'schedules.nbf': {'$lte': now},\n        'schedules.exp': {'$gte': now}\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'username': True,\n        'tag': '$schedules.tag',\n        'nbf': '$schedules.nbf',\n        'exp': '$schedules.exp'\n    }}\n    yield {'$group': {\n        '_id': '$id',\n        'tags': {'$addToSet': '$tag'},\n    }}\n\nfor user_tag_schedule in User.objects.aggregate(*stages()):\n    print(user_tag_schedule)\n\n# output:\n# {'_id': ObjectId('579b9387b7af8e1fd1635da9'), 'tags': ['stats']}\n# {'_id': ObjectId('5a66d5c2af9c462c617ce552'), 'tags': ['chat', 'vip']}<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/group\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/group\/<\/a><\/p>\n<h3>Project A New Field Based On Whether Elements Exist In Another Array Field<\/h3>\n<p>Use <code>$addFields<\/code> with <code>$cond<\/code>.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    user_preferences = g.user.settings.preferences or ['gender:female']\n    yield {'$match': {\n        'gender': {'$in': [prefix_gender.replace('gender:', '') for prefix_gender in user_preferences]}\n    }}\n\n    yield {'$addFields': {\n        'isPinned': {'$cond': {\n            'if': {'$in': [constants.tags.HIDDEN, '$badges']},\n            'then': True,\n            'else': False,\n        }},\n    }}\n    yield {'$sort': {\n        'isPinned': -1,\n        'posted_at': -1,\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'author': '$author',\n        'title': '$title',\n        'location': '$location',\n        'postedAt': {'$floor': {'$divide': [{'$subtract': ['$posted_at', constants.UNIX_EPOCH]}, 1000]}},\n        'viewCount': '$view_count',\n        'commentCount': {'$size': {'$ifNull': ['$comments', []]}},\n        'badges': '$badges',\n        'isPinned': '$isPinned',\n    }}\n\ntry:\n    results = Post.objects.aggregate(*stages()).next()\nexcept StopIteration:\n    return Response(status=http.HTTPStatus.NOT_FOUND)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/16512329\/project-new-boolean-field-based-on-element-exists-in-an-array-of-a-subdocument\">https:\/\/stackoverflow.com\/questions\/16512329\/project-new-boolean-field-based-on-element-exists-in-an-array-of-a-subdocument<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/project\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/project\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/addFields\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/addFields\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/cond\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/cond\/<\/a><\/p>\n<h3>Project And Filter Out Elements Of An Array With <code>$filter<\/code><\/h3>\n<p>Elements in <code>details<\/code> might have no <code>value<\/code> field.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    yield {'$match': {\n        '_id': bson.ObjectId(post_id),\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'author': '$author',\n        'title': '$title',\n        'location': '$location',\n        'postedAt': {'$floor': {'$divide': [{'$subtract': ['$posted_at', constants.UNIX_EPOCH]}, 1000]}},\n        'viewCount': '$view_count',\n        'commentCount': {'$size': '$comments'},\n        'details': [\n            {'key': 'gender', 'value': '$gender'},\n            {'key': 'pricing', 'value': '$pricing'},\n            {'key': 'lineId', 'value': {'$ifNull': ['$lineId', None]}},\n            {'key': 'description', 'value': {'$ifNull': ['$description', None]}},\n        ],\n    }}\n    yield {'$addFields': {\n        'details': {\n            '$filter': {\n                'input': '$details',\n                'as': 'detail',\n                'cond': {'$ne': ['$$detail.value', None]},\n            }\n        }\n    }}\n\ntry:\n    post = next(Post.objects.aggregate(*stages()))\nexcept StopIteration:\n    return Response(status=http.HTTPStatus.NOT_FOUND)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/filter\/#exp._S_filter\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/filter\/#exp._S_filter<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/addFields\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/addFields\/<\/a><\/p>\n<h3>Project Specific Fields Of Elements Of An Array With <code>$map<\/code><\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    yield {'$match': {\n        '_id': bson.ObjectId(post_id),\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'author': '$author',\n        'title': '$title',\n        'location': '$location',\n        'postedAt': {'$floor': {'$divide': [{'$subtract': ['$posted_at', constants.UNIX_EPOCH]}, 1000]}},\n        'viewCount': '$view_count',\n        'commentCount': {'$size': '$comments'},\n        'details': [\n            {'key': 'gender', 'value': '$gender'},\n            {'key': 'pricing', 'value': '$pricing'},\n            {'key': 'lineId', 'value': {'$ifNull': ['$lineId', None]}},\n            {'key': 'description', 'value': {'$ifNull': ['$description', None]}},\n        ],\n        'media': {\n            '$map': {\n                'input': '$media',\n                'as': 'transcoded_media',\n                'in': {\n                    'mimetype': '$$transcoded_media.mimetype',\n                    'dash': '$$transcoded_media.presets.dash',\n                    'hls': '$$transcoded_media.presets.hls',\n                    'thumbnail': '$$transcoded_media.thumbnail',\n                }\n            }\n        },\n    }}\n    yield {'$addFields': {\n        'details': {\n            '$filter': {\n                'input': '$details',\n                'as': 'detail',\n                'cond': {'$ne': ['$$detail.value', None]},\n            }\n        }\n    }}\n\ntry:\n    post = next(Post.objects.aggregate(*stages()))\nexcept StopIteration:\n    return Response(status=http.HTTPStatus.NOT_FOUND)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/33831665\/how-to-project-specific-fields-from-a-document-inside-an-array\">https:\/\/stackoverflow.com\/questions\/33831665\/how-to-project-specific-fields-from-a-document-inside-an-array<\/a><\/p>\n<h3>Do Advanced <code>$project<\/code> With <code>$let<\/code><\/h3>\n<p>If you find yourself wanting to do <code>$project<\/code> twice to tackle some fields, you should use <code>$let<\/code>.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    yield {'$match': {\n        'purchases.user': g.user.id,\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'name': True,\n        'image': {\n            '$ifNull': [{'$arrayElemAt': ['$images', 0]}, None],\n        },\n        'purchasedAt': {\n            '$let': {\n                'vars': {\n                    'purchase': {\n                        '$arrayElemAt': [\n                            {\n                                '$filter': {\n                                    'input': '$purchases',\n                                    'as': 'purchase',\n                                    'cond': {\n                                        '$and': [\n                                            {'$eq': ['$$purchase.user', g.user.id]},\n                                        ],\n                                    },\n                                },\n                            },\n                            0,\n                        ],\n                    },\n                },\n                'in': '$$purchase.timestamp',\n            },\n        },\n    }}\n\ntry:\n    docs = MessagePackProduct.objects.aggregate(*stages())\nexcept StopIteration:\n    docs = []\nelse:\n    for doc in docs:\n        print(doc)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/let\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/let\/<\/a><\/p>\n<h3>Deconstruct An Array Field With <code>$unwind<\/code> And Query Them With <code>$match<\/code><\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    category_tag = 'category:user'\n    currency = 'usd'\n    platform = 'ios'\n\n    yield {'$match': {\n        'active': True,\n        'tags': category_tag,\n        'total': {'$gt': 0},\n        'preview_message': {'$exists': True},\n    }}\n    yield {'$unwind': '$skus'}\n    yield {'$match': {\n        'skus.attributes.platform': platform,\n        'skus.attributes.currency': currency,\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'name': True,\n        'caption': True,\n        'description': True,\n        'image': {\n            '$ifNull': [{'$arrayElemAt': ['$images', 0]}, None],\n        },\n        'sku': '$skus',\n        'created_at': True,\n        'is_purchased': {'$in': [g.user.id, {'$ifNull': ['$purchases.user', []]}]},\n    }}\n    yield {'$sort': {'is_purchased': 1, 'created_at': -1}}\n\ntry:\n    docs = MessagePackProduct.objects.aggregate(*stages())\nexcept StopIteration:\n    docs = []\nelse:\n    for doc in docs:\n        print(doc)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/match\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/match\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/unwind\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/unwind\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/project\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/project\/<\/a><\/p>\n<h3>Query The First Element In An Array Field With <code>$arrayElemAt<\/code> And <code>$filter<\/code><\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    category_tag = 'category:user'\n    currency = 'usd'\n    platform = 'ios'\n\n    yield {'$match': {\n        'active': True,\n        'tags': category_tag,\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'name': True,\n        'caption': True,\n        'description': True,\n        'image': {\n            '$ifNull': [{'$arrayElemAt': ['$images', 0]}, None],\n        },\n        'preview_message': True,\n        'metadata': True,\n        'created_at': True,\n        'updated_at': True,\n        'active': True,\n        'sku': {\n            '$ifNull': [\n                {\n                    '$arrayElemAt': [\n                        {\n                            '$filter': {\n                                'input': '$skus',\n                                'as': 'sku',\n                                'cond': {\n                                    '$and': [\n                                        {'$eq': ['$$sku.currency', currency]},\n                                        {'$eq': ['$$sku.attributes.platform', platform]},\n                                    ]\n                                }\n                            },\n                        },\n                        0\n                    ]\n                },\n                None\n            ],\n        },\n        'tags': True,\n        'total': True,\n        'is_bought': {'$in': [g.user.id, {'$ifNull': ['$purchases.user', []]}]},\n    }}\n    yield {'$sort': {'is_bought': 1, 'created_at': -1}}\n\ntry:\n    docs = MessagePackProduct.objects.aggregate(*stages())\nexcept StopIteration:\n    docs = []\nelse:\n    for doc in docs:\n        print(doc)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/master\/reference\/operator\/aggregation\/filter\/\">https:\/\/docs.mongodb.com\/master\/reference\/operator\/aggregation\/filter\/<\/a><br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/3985214\/retrieve-only-the-queried-element-in-an-object-array-in-mongodb-collection\">https:\/\/stackoverflow.com\/questions\/3985214\/retrieve-only-the-queried-element-in-an-object-array-in-mongodb-collection<\/a><\/p>\n<h3>Join Another Collection Using <code>$lookup<\/code><\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    yield {'$match': {\n        'tags': 'pack:prod_CR1u34BIpDbHeo',\n    }}\n    yield {'$lookup': {\n        'from': 'user',\n        'localField': 'sender',\n        'foreignField': '_id',\n        'as': 'sender_data',\n    }}\n    yield {'$unwind': '$sender_data'}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'sender': {\n            'id': '$sender_data._id',\n            'username': '$sender_data.username',\n        },\n        'caption': True,\n        'posted_at': True,\n    }}\n    yield {'$sort': {'posted_at': -1}}\n\ntry:\n    docs = Message.objects.aggregate(*stages())\nexcept StopIteration:\n    docs = []\nelse:\n    for doc in docs:\n        print(doc)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/lookup\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/lookup\/<\/a><br \/>\n<a href=\"https:\/\/thecodebarbarian.com\/a-nodejs-perspective-on-mongodb-36-lookup-expr\">https:\/\/thecodebarbarian.com\/a-nodejs-perspective-on-mongodb-36-lookup-expr<\/a><\/p>\n<h3>Join Another Collection With Multiple Conditions Using <code>pipeline<\/code> in <code>$lookup<\/code><\/h3>\n<p>To access the <code>let<\/code> variables in the <code>$lookup<\/code> pipeline, you could only use the <code>$expr<\/code> operator.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">var start = ISODate('2018-09-22T00:00:00.000+08:00');\n\ndb.getCollection('feature.shop.order').aggregate([\n    {'$match': {\n        'payment.timestamp': {'$gte': start},\n        'status': {'$in': ['paid']},\n    }},\n    {'$lookup': {\n        'from': 'user',\n        'localField': 'customer',\n        'foreignField': '_id',\n        'as': 'customer_data',\n    }},\n    {'$unwind': '$customer_data'},\n    {'$project': {\n        'variation': '$customer_data.experiments.message_unlock_price.variation',\n        'amount_normalized': {'$divide': ['$amount', 100.0]},\n    }},\n    {'$addFields': {\n        'amount_usd': {'$multiply': ['$amount_normalized', 0.033]},\n    }},\n    {'$group': {\n       '_id': '$variation',\n       'purchase_amount': {'$sum': '$amount_usd'},\n       'paid_user_count': {'$sum': 1},\n    }},\n    {'$lookup': {\n        'from': 'user',\n        'let': {\n            'variation': '$_id',\n        },\n        'pipeline': [\n            {'$match': {\n                'last_active': {'$gte': start},\n                'experiments': {'$exists': true},\n            }},\n            {'$match': {\n                '$expr': {\n                    '$and': [\n                         {'$eq': ['$experiments.message_unlock_price.variation', '$$variation']},\n                    ],\n                },\n            }},\n            {'$group': {\n               '_id': '$experiments.message_unlock_price.variation',\n               'count': {'$sum': 1},\n            }},\n        ],\n        'as': 'variation_data',\n    }},\n    {'$unwind': '$variation_data'},\n    {'$project': {\n        '_id': 1,\n        'purchase_amount': 1,\n        'paid_user_count': 1,\n        'total_user_count': '$variation_data.count',\n    }},\n    {'$addFields': {\n        'since': start,\n        'arpu': {'$divide': ['$purchase_amount', '$total_user_count']},\n        'arppu': {'$divide': ['$purchase_amount', '$paid_user_count']},\n    }},\n    {'$sort': {'_id': 1}},\n]);<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/lookup\/#join-conditions-and-uncorrelated-sub-queries\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/lookup\/#join-conditions-and-uncorrelated-sub-queries<\/a><\/p>\n<p>or<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    yield {'$match': {'_id': bson.ObjectId(message_id)}}\n    yield {'$limit': 1}\n    yield {'$project': {\n        '_cls': 1,\n        'sender': 1,\n        'unlocks': 1,\n    }}\n    yield {'$unwind': '$unlocks'}\n    yield {'$match': {\n        'unlocks.user': bson.ObjectId(user_id),\n        'unlocks.amount': {'$gt': 0},\n    }}\n    yield {'$lookup': {\n        'from': 'user',\n        'let': {\n            'sender': '$sender',\n            'unlocker': '$unlocks.user',\n        },\n        'pipeline': [\n            {'$match': {\n                '$expr': {\n                    '$or': [\n                        {'$eq': ['$_id', '$$sender']},\n                        {'$eq': ['$_id', '$$unlocker']}\n                    ]\n                }\n            }}\n        ],\n        'as': 'users',\n    }}\n    yield {'$addFields': {\n        'sender': {'$arrayElemAt': ['$users', 0]},\n        'unlocker': {'$arrayElemAt': ['$users', 1]},\n    }},\n    yield {'$project': {\n        '_id': 0,\n        '_cls': 1,\n        'id': '$_id',\n        'sender': {\n            'id': '$sender._id',\n            'username': '$sender.username',\n        },\n        'unlocker': {\n            'id': '$unlocker._id',\n            'username': '$unlocker.username',\n        },\n        'amount': '$unlocks.amount',\n    }}\n\ntry:\n    context = Message.objects.aggregate(*stages()).next()\nexcept StopIteration:\n    pass<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/37086387\/multiple-join-conditions-using-the-lookup-operator\">https:\/\/stackoverflow.com\/questions\/37086387\/multiple-join-conditions-using-the-lookup-operator<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/lookup\/#specify-multiple-join-conditions-with-lookup\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/lookup\/#specify-multiple-join-conditions-with-lookup<\/a><\/p>\n<h3>Count Documents In Another Collection With <code>$lookup<\/code> (JOIN)<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    category_tag = f'category:{category}'\n    yield {'$match': {\n        'active': True,\n        'tags': category_tag,\n    }}\n    yield {'$addFields': {\n        'message_pack_id_tag': {'$concat': ['pack:', '$_id']},\n    }}\n    yield {'$lookup': {\n        'from': 'message',\n        'localField': 'message_pack_id_tag',\n        'foreignField': 'tags',\n        'as': 'total',\n    }}\n    yield {'$addFields': {\n        'total': {'$size': '$total'}\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'name': True,\n        'total': True,\n    }}\n\ntry:\n    docs = MessagePackProduct.objects.aggregate(*stages())\nexcept StopIteration:\n    docs = []\nelse:\n    for doc in docs:\n        print(doc)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/lookup\/#equality-match\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/lookup\/#equality-match<\/a><\/p>\n<h3>Use <code>$lookup<\/code> as <code>findOne()<\/code> Which Returns An Object<\/h3>\n<p>Use <code>$lookup<\/code> and <code>$unwind<\/code>.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-py\">import bson\n\ndef stages():\n    yield {'$match': {'_id': bson.ObjectId(gift_id)}}\n    yield {'$limit': 1}\n    yield {'$lookup': {\n        'from': 'user',\n        'localField': 'sender',\n        'foreignField': '_id',\n        'as': 'sender',\n    }}\n    yield {'$unwind': '$sender'}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'sender': {\n            'id': '$sender._id',\n            'username': '$sender.username',\n        },\n        'product_id': '$product._id',\n        'sent_at': '$sent_at',\n        'amount': '$cost.amount',\n    }}\n\ntry:\n    _context = Gift.objects.aggregate(*stages()).next()\nexcept StopIteration:\n    pass<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/37691727\/how-to-use-mongodbs-aggregate-lookup-as-findone\">https:\/\/stackoverflow.com\/questions\/37691727\/how-to-use-mongodbs-aggregate-lookup-as-findone<\/a><\/p>\n<h3>Collapse Documents In An Array<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    yield {'$match': {\n        'tags': f'tutorial:buy-diamonds:v1',\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'caption.text': True,\n        'sender': True,\n        'media.type': '$media.mimetype',\n    }}\n    yield {'$facet': {\n        'inbox': [\n            {'$sort': {'created_at': -1}},\n            {'$limit': 10}\n        ],\n    }}\n    yield {'$project': {\n        'inbox': True,\n        'required_unlock_count': {'$literal': 5},\n        'price_per_message': {'$literal': 1200},\n    }}\n\ntry:\n    result = Message.objects.aggregate(*stages()).next()\nexcept StopIteration:\n    result = {}<\/code><\/pre>\n<p>JSON output:<\/p>\n<pre class=\"line-numbers\"><code class=\"language-json\">{\n    \"inbox\": [\n        {\n            \"caption\": {\n                \"text\": \"fuck yeah\"\n            },\n            \"id\": \"5aaba1e9593950337a90dcb3\",\n            \"media\": {\n                \"type\": \"video\/mp4\"\n            },\n            \"sender\": \"5a66d5c2af9c462c617ce552\"\n        },\n        {\n            \"caption\": {\n                \"text\": \"test\"\n            },\n            \"id\": \"5ad549276b2c362a4efe5e21\",\n            \"media\": {\n                \"type\": \"image\/jpeg\"\n            },\n            \"sender\": \"5a66d5c2af9c462c617ce552\"\n        }\n    ],\n    \"price_per_message\": 1200,\n    \"required_unlock_count\": 5\n}<\/code><\/pre>\n<h3>Do Pagination With <code>$facet<\/code> And <code>$project<\/code><\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    # normal query\n    yield {'$match': {\n        'purchases.user': g.user.id,\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'name': True,\n        'created_at': True,\n        'meta': {\n            'revision': '$revision',\n            'tags': '$tags',\n        },\n    }}\n    yield {'$sort': {'created_at': -1}}\n\n    # pagination\n    page = 0\n    limit = 10\n    yield {'$facet': {\n        'meta': [\n            {'$count': 'total'},\n        ],\n        'objects': [\n            {'$skip': page * limit},\n            {'$limit': limit},\n        ]\n    }}\n    # JSON output:\n    # {\n    #    \"meta\": [\n    #       {\"total\": 2}\n    #    ],\n    #    \"objects\": [\n    #       {\n    #          \"id\": \"prod_CR1u34BIpDbHeo\",\n    #          \"name\": \"Product Name 2\"\n    #       },\n    #       {\n    #          \"id\": \"prod_Fkhf9JFK3Rdgk9\",\n    #          \"name\": \"Product Name 1\"\n    #       }\n    #    ]\n    # }\n    yield {'$project': {\n        'total': {'$let': {\n            'vars': {\n                'meta': {'$arrayElemAt': ['$meta', 0]},\n            },\n            'in': '$$meta.total',\n        }},\n        'objects': True,\n    }}\n    # JSON output:\n    # {\n    #    \"total\": 2,\n    #    \"objects\": [\n    #       {\n    #          \"id\": \"prod_CR1u34BIpDbHeo\",\n    #          \"name\": \"Product Name 2\"\n    #       },\n    #       {\n    #          \"id\": \"prod_Fkhf9JFK3Rdgk9\",\n    #          \"name\": \"Product Name 1\"\n    #       }\n    #    ]\n    # }\n\ntry:\n    output = MessagePackProduct.objects.aggregate(*stages()).next()\nexcept StopIteration:\n    output = {}\nelse:\n    print(output)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/facet\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/facet\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/project\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/project\/<\/a><\/p>\n<h3>Perform <code>$facet<\/code> + <code>$project<\/code> =&gt; Unwrap with <code>$unwind<\/code> =&gt; Do <code>$facet<\/code> + <code>$project<\/code> Again<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    yield {'$match': {\n        'purchases.user': g.user.id,\n    }}\n    yield {'$project': {\n        '_id': False,\n        'id': '$_id',\n        'name': True,\n        'image': {\n            '$ifNull': [{'$arrayElemAt': ['$images', 0]}, None],\n        },\n        'created_at': True,\n    }}\n    yield {'$sort': {'created_at': -1}}\n\n    # pagination\n    page = 0\n    limit = 10\n    yield {'$facet': {\n        'meta': [\n            {'$count': 'total'},\n        ],\n        'objects': [\n            {'$skip': page * limit},\n            {'$limit': limit},\n        ]\n    }}\n    yield {'$project': {\n        'total': {'$let': {\n            'vars': {\n                'meta': {'$arrayElemAt': ['$meta', 0]},\n            },\n            'in': '$$meta.total',\n        }},\n        'objects': True,\n    }}\n\n    # do $lookup after the pagination\n    yield {'$unwind': '$objects'}\n    yield {'$addFields': {\n        'objects.message_pack_id_tag': {'$concat': ['pack:', '$objects.id']},\n    }}\n    yield {'$lookup': {\n        'from': 'message',\n        'localField': 'objects.message_pack_id_tag',\n        'foreignField': 'tags',\n        'as': 'objects.total',\n    }}\n    yield {'$addFields': {\n        'objects.total': {'$size': '$objects.total'}\n    }}\n\n    # re-wrap into the pagination structure\n    yield {'$facet': {\n        'total_list': [\n            {'$project': {\n                'total': True,\n            }},\n        ],\n        'objects': [\n            {'$replaceRoot': {'newRoot': '$objects'}},\n        ]\n    }}\n    yield {'$project': {\n        'total': {'$let': {\n            'vars': {\n                'meta': {'$arrayElemAt': ['$total_list', 0]},\n            },\n            'in': '$$meta.total',\n        }},\n        'objects': True,\n    }}\n\ntry:\n    output = MessagePackProduct.objects.aggregate(*stages()).next()\nexcept StopIteration:\n    output = {}\nelse:\n    print(output)<\/code><\/pre>\n<h3>Do <code>$group<\/code> First To Reduce Numbers Of <code>$lookup<\/code> Calls<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-py\">def stages():\n    yield {'$match': {\n        'tags': f'pack:{message_pack_id}',\n    }}\n    yield {'$group': {\n        '_id': '$sender',\n        'messages': {'$push': '$$ROOT'},\n    }}\n    yield {'$lookup': {\n        'from': 'user',\n        'localField': '_id',\n        'foreignField': '_id',\n        'as': 'sender_data',\n    }}\n    yield {'$unwind': '$messages'}\n    yield {'$project': {\n        '_id': False,\n        'id': '$messages._id',\n        'caption': {\n            'text': '$messages.caption.text',\n            'y': '$messages.caption.y',\n        },\n        'sender': {\n            'id': {'$arrayElemAt': ['$sender_data._id', 0]},\n            'username': {'$arrayElemAt': ['$sender_data.username', 0]},\n        },\n    }}\n\ntry:\n    docs = Message.objects.aggregate(*stages())\nexcept StopIteration:\n    docs = []\nelse:\n    for doc in docs:\n        print(doc)<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/group\/\">https:\/\/docs.mongodb.com\/manual\/reference\/operator\/aggregation\/group\/<\/a><\/p>\n<h3>Copy Collections To Another Database<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-js\">var bulk = db.getSiblingDB('target_db')['target_collection'].initializeOrderedBulkOp();\ndb.getCollection('source_collection').find().forEach(function(d) {\n    bulk.insert(d);\n});\nbulk.execute();\n\nvar bulk = db.getSiblingDB('test')['company.revenue'].initializeOrderedBulkOp();\ndb.getCollection('company.revenue').find().forEach(function(d) {\n    bulk.insert(d);\n});\nbulk.execute();\n\nvar bulk = db.getSiblingDB('test')['user.contract'].initializeOrderedBulkOp();\ndb.getCollection('user.contract').find().forEach(function(d) {\n    bulk.insert(d);\n});\nbulk.execute();\n\nvar bulk = db.getSiblingDB('test')['user.revenue'].initializeOrderedBulkOp();\ndb.getCollection('user.revenue').find().forEach(function(d) {\n    bulk.insert(d);\n});\nbulk.execute();<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/11554762\/how-to-copy-a-collection-from-one-database-to-another-in-mongodb\">https:\/\/stackoverflow.com\/questions\/11554762\/how-to-copy-a-collection-from-one-database-to-another-in-mongodb<\/a><\/p>\n<p>Sadly, <code>cloneCollection()<\/code> cannot clone collections from one local database to another local database.<\/p>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/command\/cloneCollection\/\">https:\/\/docs.mongodb.com\/manual\/reference\/command\/cloneCollection\/<\/a><\/p>\n<h2>Useful Tools<\/h2>\n<h3>Backup<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-console\">$ mongodump -h  127.0.0.1:27017 --oplog -j=8 --gzip --archive=\/data\/mongodump.tar.gz<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/program\/mongodump\/\">https:\/\/docs.mongodb.com\/manual\/reference\/program\/mongodump\/<\/a><\/p>\n<h3>Restore<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-console\">$ mongorestore --drop --gzip --archive=2018-08-12T03.tar.gz<\/code><\/pre>\n<p>This kind of error typically indicates some sort of issue with data corruption, which is often caused by problems with the underlying storage device, file system or network connection.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-console\">restoring indexes for collection test.message from metadata\nFailed: test.message: error creating indexes for test.message: createIndex error: BSONElement: bad type -47<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/program\/mongorestore\/\">https:\/\/docs.mongodb.com\/manual\/reference\/program\/mongorestore\/<\/a><\/p>\n<h3>Profiling<\/h3>\n<p>You could also set the profiling level to <code>2<\/code> to record every query.<\/p>\n<pre class=\"line-numbers\"><code class=\"language-js\">db.setProfilingLevel(2);\n\ndb.getCollection('system.profile').find({\n    'ns': { \n        '$nin' : ['test.system.profile', 'test.system.indexes', 'test.system.js', 'test.system.users']\n    }\n}).limit(5).sort({'ts': -1}).pretty();<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/tutorial\/manage-the-database-profiler\/\">https:\/\/docs.mongodb.com\/manual\/tutorial\/manage-the-database-profiler\/<\/a><br \/>\n<a href=\"https:\/\/stackoverflow.com\/questions\/15204341\/mongodb-logging-all-queries\">https:\/\/stackoverflow.com\/questions\/15204341\/mongodb-logging-all-queries<\/a><\/p>\n<pre class=\"line-numbers\"><code class=\"language-console\">$ pip install mongotail\n\n# set the profiling level\n$ mongotail 127.0.0.1:27017\/test -l 2\n\n# tail logs\n$ mongotail 127.0.0.1:27017\/test -f -m -f<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/github.com\/mrsarm\/mongotail\">https:\/\/github.com\/mrsarm\/mongotail<\/a><\/p>\n<h3>Monitoring<\/h3>\n<pre class=\"line-numbers\"><code class=\"language-console\">$ mongotop\n$ mongostat<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/program\/mongotop\/\">https:\/\/docs.mongodb.com\/manual\/reference\/program\/mongotop\/<\/a><br \/>\n<a href=\"https:\/\/docs.mongodb.com\/manual\/reference\/program\/mongostat\/\">https:\/\/docs.mongodb.com\/manual\/reference\/program\/mongostat\/<\/a><\/p>\n<pre class=\"line-numbers\"><code class=\"language-console\">$ pip install mtools\n\n$ mloginfo mongod.log<\/code><\/pre>\n<p>ref:<br \/>\n<a href=\"https:\/\/github.com\/rueckstiess\/mtools\">https:\/\/github.com\/rueckstiess\/mtools<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Frequently accessed items are cached in memory, so that MongoDB can provide optimal response time.<\/p>\n","protected":false},"author":1,"featured_media":513,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[22,7,4],"tags":[11,119,2],"class_list":["post-512","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-about-database","category-about-javascript","category-about-python","tag-javascript","tag-mongodb","tag-python"],"_links":{"self":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/posts\/512","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/comments?post=512"}],"version-history":[{"count":0,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/posts\/512\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/media\/513"}],"wp:attachment":[{"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/media?parent=512"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/categories?post=512"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/vinta.ws\/code\/wp-json\/wp\/v2\/tags?post=512"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}