Frequently accessed items are cached in memory, so that MongoDB can provide optimal response time.
MongoDB Shell in JavaScript
Administration
ref:
https://docs.mongodb.com/manual/reference/method/db.currentOp/
https://hackernoon.com/mongodb-currentop-18fe2f9dbd68
http://www.mongoing.com/archives/6246
BSON Types
ref:
https://docs.mongodb.com/manual/reference/bson-types/
Check If A Document Exists
It is significantly faster to use find()
+ limit()
because findOne()
will always read + return the document if it exists. find()
just returns a cursor (or not) and only reads the data if you iterate through the cursor.
ref:
https://stackoverflow.com/questions/8389811/how-to-query-mongodb-to-test-if-an-item-exists
https://blog.serverdensity.com/checking-if-a-document-exists-mongodb-slow-findone-vs-find/
Find Documents
Find Documents With Regular Expression
ref:
https://docs.mongodb.com/manual/reference/operator/query/regex/
Find Documents With An Array Field
$in: [...]
means "intersection" or "any element in"
$all: [...]
means "subset" or "contain"
$elemMatch: {...}
means "any element match"
$not: {$elemMatch: {$nin: [...]}}
means "subset" or "in"
The last one roughly means not any([False, False, False, False])
where each False
is indicating if the item is not in in [...]
.
ref:
https://stackoverflow.com/questions/12223465/mongodb-query-subset-of-an-array
ref:
https://docs.mongodb.com/manual/reference/operator/query/exists/#exists-true
Find Documents With An Array Field Of Embedded Documents
Usually, you could use $elemMatch
.
ref:
https://docs.mongodb.com/manual/reference/operator/query/elemMatch/
Find Documents With Existence Of Fields Or Values
.find({'field': {'$exists': true}})
: the field exists
.find({'field': {'$exists': false}})
: the field does not exist
.find({'field': {'$type': 10}})
: the field exists with a null
value
.find({'field': null})
: the field exists with a null
value or the field does not exist
.find({'field': {'$ne': null}})
: the field exists and the value is not null
.find({'array_field': {'$in': [null, []]}})
ref:
https://stackoverflow.com/questions/4057196/how-do-you-query-this-in-mongo-is-not-null
https://docs.mongodb.com/manual/tutorial/query-for-null-fields/
Find Documents Where An Array Field Does Not Contain A Certain Value
ref:
https://stackoverflow.com/questions/16221599/find-documents-with-arrays-not-containing-a-document-with-a-particular-field-val
Find Documents Where An Array Field Is Not Empty
ref:
https://stackoverflow.com/questions/14789684/find-mongodb-records-where-array-field-is-not-empty
Find Documents Where An Array Field's Size Is Greater Than 1
ref:
https://stackoverflow.com/questions/7811163/query-for-documents-where-array-size-is-greater-than-1/15224544
Find Documents With Computed Values Using $expr
For instance, compare 2 fields from a single document in a find()
query.
ref:
https://thecodebarbarian.com/a-nodejs-perspective-on-mongodb-36-lookup-expr
https://dzone.com/articles/expressive-query-language-in-mongodb-36-2
Project A Subset Of An Array Field With $filter
A sample document:
ref:
https://stackoverflow.com/questions/42607221/mongodb-aggregation-project-check-if-array-contains
Insert Documents
Update Within A For Loop
Update With Conditions Of Field Values
You could update the value of the field to a specified value if the specified value is less than or greater than the current value of the field. The $min
and $max
operators can compare values of different types.
Only set posted_at
to current timestamp if its current value is None or absent.
ref:
https://docs.mongodb.com/manual/reference/operator/update/min/
https://docs.mongodb.com/manual/reference/operator/update/max/
Update An Array Field
Array update operators:
$
: Acts as a placeholder to update the first element in an array for documents that matches the query condition.
$[]
: Acts as a placeholder to update all elements in an array for documents that match the query condition.
$[<identifier>]
: Acts as a placeholder to update elements in an array that match the arrayFilters
condition.
$addToSet
: Adds elements to an array only if they do not already exist in the set.
$push
: Adds an item to an array.
$pop
: Removes the first or last item of an array.
$pull
: Removes all array elements that match a specified query.
$pullAll
: Removes all matching values from an array.
ref:
https://docs.mongodb.com/manual/reference/operator/update-array/
http://docs.mongoengine.org/guide/querying.html#atomic-updates
http://thecodebarbarian.com/a-nodejs-perspective-on-mongodb-36-array-filters.html
Add an element in an array field.
Insert an element into an array at a certain position.
ref:
https://docs.mongodb.com/manual/reference/operator/update/position/
http://docs.mongoengine.org/guide/querying.html#querying-lists
Remove elements in an array field. It is also worth noting that update(pull__abc=xyz)
always returns 1
.
Remove multiple embedded documents in an array field.
ref:
https://stackoverflow.com/questions/28102691/pullall-while-removing-embedded-objects
ref:
https://docs.mongodb.com/manual/reference/operator/update/pull/
You could also use add_to_set
to add an item to an array only if it is not in the list, which always returns 1
if filter()
matches any document. However, you are able to set full_result=True
to get detail updated result.
ref:
http://docs.mongoengine.org/guide/querying.html#atomic-updates
Update a multi-level nested array field. Yes, arrayFilters
supports it.
ref:
https://docs.mongodb.com/manual/reference/operator/update/positional-filtered/
https://stackoverflow.com/questions/23577123/updating-a-nested-array-with-mongodb
Update an embedding document in an array field.
ref:
https://stackoverflow.com/questions/9200399/replacing-embedded-document-in-array-in-mongodb
https://docs.mongodb.com/manual/reference/method/db.collection.update/#db.collection.update
Update specific embedded documents with arrayFilters
in an array field.
User data:
It is worth noting that <identifier>
in $arrayFilters
can only contain lowercase alphanumeric characters.
ref:
https://docs.mongodb.com/master/reference/operator/update/positional-filtered/
Update An Array Field With arrayFilters
You should use arrayFilters
as much as possible.
The syntax of arrayFilters
would be:
ref:
https://docs.mongodb.com/manual/reference/operator/update/positional-filtered/
Insert an element into an array field at a certain position.
Or use explicit array index $set
.
ref:
https://docs.mongodb.com/manual/reference/operator/update/position/
Set an array field to empty.
ref:
https://docs.mongodb.com/manual/reference/method/db.collection.update/
https://docs.mongodb.com/manual/reference/operator/update/set/
Remove elements from an array field.
ref:
https://docs.mongodb.com/manual/reference/operator/update/pull/
Update A Dictionary Field
Set a key/value in a dictionary field.
ref:
https://stackoverflow.com/questions/21158028/updating-a-dictfield-in-mongoengine
Upsert: Update Or Create
You must use upsert=true
with uniquely indexed fields. If you don't need the modified document, you should just use update_one(field1=123, field2=456, upsert=True)
.
Additionally, remember that modify()
always reloads the whole object even the original one only loads specific fields with only()
. Try to avoid using document.DB_QUERY_METHOD()
, and using User.objects.filter().only().modify()
or User.objects.filter().update()
when it is possible.
ref:
https://docs.mongodb.com/manual/reference/method/db.collection.update/#update-with-unique-indexes
http://docs.mongoengine.org/apireference.html#mongoengine.queryset.QuerySet.modify
http://docs.mongoengine.org/apireference.html#mongoengine.queryset.QuerySet.update_one
Rename A Field
Simply rename a field with $rename
.
ref:
https://docs.mongodb.com/manual/reference/operator/update/rename/
Do some extra data converting and rename the field manually.
Insert/Replace Large Amount Of Documents
Update Large Numbers Of Documents
Use Bulk.find.arrayFilters()
and Bulk.find.update()
together.
In Python:
In JavaScript:
ref:
https://docs.mongodb.com/manual/reference/method/Bulk/
https://docs.mongodb.com/manual/reference/method/Bulk.find.arrayFilters/
Of course, you could also update the same document with multiple operations. However, it does not make sense.
ref:
https://api.mongodb.com/python/current/examples/bulk.html
Remove items from an array field of documents.
ref:
https://stackoverflow.com/questions/33594397/how-to-update-a-large-number-of-documents-in-mongodb-most-effeciently
Remove Large Numbers Of Documents
in mongo
shell:
ref:
https://docs.mongodb.com/manual/reference/method/Bulk.find.remove/#bulk-find-remove
MongoEngine In Python
ref:
http://docs.mongoengine.org/guide/index.html
http://docs.mongoengine.org/apireference.html
Define Collections
It seems every collection in MongoEngine must have a id
field.
ref:
http://docs.mongoengine.org/guide/defining-documents.html
Define A Field With Default EmbeddedDocument
The behavior of setting an EmbeddedDocument
class as default works differently with and without only()
.
If the user does not have settings
field in DB, here is the difference.
Filter With Raw Queries
ref:
http://docs.mongoengine.org/guide/querying.html#raw-queries
Check If A Document Exists
Use .exists()
.
You have to use __raw__
if the field you want to query is a db.ListField(GenericEmbeddedDocumentField(XXX)
field.
Upsert: Get Or Create
ref:
http://docs.mongoengine.org/apireference.html#mongoengine.queryset.QuerySet.upsert_one
Store Files On GridFS
ref:
http://docs.mongoengine.org/guide/gridfs.html
Store Datetime
MongoDB stores datetimes in UTC.
ref:
https://docs.mongodb.com/manual/reference/method/Date/
2-phase Commit
The easiest way to think about 2-phase commit is idempotency, i.e., if you run a update many times, the results would "be the same": initial -> pending -> applied -> done.
ref:
https://docs.mongodb.com/manual/tutorial/perform-two-phase-commits/
Aggregation Pipeline
$match
: Filters documents.
$project
: Modifies document fields.
$addFields
: Adds or overrides document fields.
$group
: Groups documents by fields.
$lookup
: Joins another collection.
$replaceRoot
: Promotes an embedded document field to the top level and replace all other fields.
$unwind
: Expanses an array field into multiple documents along with original documents.
$facet
: Processes multiple pipelines within one stage and output to different fields.
There are special system variables, for instance, $$ROOT
, $$REMOVE
, $$PRUNE
, which you could use in some stages of the aggregation pipeline.
ref:
https://docs.mongodb.com/manual/reference/aggregation-variables/#system-variables
Return Date As Unix Timestamp
ref:
https://stackoverflow.com/questions/39274311/convert-iso-date-to-timestamp-in-mongo-query
Match Multiple Conditions Which Store In An Array Fields
ref:
https://docs.mongodb.com/manual/reference/operator/query/in/
https://docs.mongodb.com/manual/reference/operator/query/nin/
https://docs.mongodb.com/manual/reference/operator/aggregation/setIsSubset/
Do Distinct With $group
ref:
https://docs.mongodb.com/manual/reference/operator/aggregation/group/
Slice Items In Each $group
Collect Items With $group
And $addToSet
User data:
ref:
https://docs.mongodb.com/manual/reference/operator/aggregation/group/
Project A New Field Based On Whether Elements Exist In Another Array Field
Use $addFields
with $cond
.
ref:
https://stackoverflow.com/questions/16512329/project-new-boolean-field-based-on-element-exists-in-an-array-of-a-subdocument
https://docs.mongodb.com/manual/reference/operator/aggregation/project/
https://docs.mongodb.com/manual/reference/operator/aggregation/addFields/
https://docs.mongodb.com/manual/reference/operator/aggregation/cond/
Project And Filter Out Elements Of An Array With $filter
Elements in details
might have no value
field.
ref:
https://docs.mongodb.com/manual/reference/operator/aggregation/filter/#exp._S_filter
https://docs.mongodb.com/manual/reference/operator/aggregation/addFields/
Project Specific Fields Of Elements Of An Array With $map
ref:
https://stackoverflow.com/questions/33831665/how-to-project-specific-fields-from-a-document-inside-an-array
Do Advanced $project
With $let
If you find youself want to do $project
twice to tackle some fields, you should use $let
.
ref:
https://docs.mongodb.com/manual/reference/operator/aggregation/let/
Deconstruct An Array Field With $unwind
And Query Them With $match
ref:
https://docs.mongodb.com/manual/reference/operator/aggregation/match/
https://docs.mongodb.com/manual/reference/operator/aggregation/unwind/
https://docs.mongodb.com/manual/reference/operator/aggregation/project/
Query The First Element In An Array Field With $arrayElemAt
And $filter
ref:
https://docs.mongodb.com/master/reference/operator/aggregation/filter/
https://stackoverflow.com/questions/3985214/retrieve-only-the-queried-element-in-an-object-array-in-mongodb-collection
Join Another Collection Using $lookup
ref:
https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/
https://thecodebarbarian.com/a-nodejs-perspective-on-mongodb-36-lookup-expr
Join Another Collection With Multiple Conditions Using pipeline
in $lookup
To access the let
variables in the $lookup
pipeline, you could only use the $expr
operator.
var start = ISODate('2018-09-22T00:00:00.000+08:00');
db.getCollection('feature.shop.order').aggregate([
{'$match': {
'payment.timestamp': {'$gte': start},
'status': {'$in': ['paid']},
}},
{'$lookup': {
'from': 'user',
'localField': 'customer',
'foreignField': '_id',
'as': 'customer_data',
}},
{'$unwind': '$customer_data'},
{'$project': {
'variation': '$customer_data.experiments.message_unlock_price.variation',
'amount_normalized': {'$divide': ['$amount', 100.0]},
}},
{'$addFields': {
'amount_usd': {'$multiply': ['$amount_normalized', 0.033]},
}},
{'$group': {
'_id': '$variation',
'purchase_amount': {'$sum': '$amount_usd'},
'paid_user_count': {'$sum': 1},
}},
{'$lookup': {
'from': 'user',
'let': {
'variation': '$_id',
},
'pipeline': [
{'$match': {
'last_active': {'$gte': start},
'experiments': {'$exists': true},
}},
{'$match': {
'$expr': {
'$and': [
{'$eq': ['$experiments.message_unlock_price.variation', '$$variation']},
],
},
}},
{'$group': {
'_id': '$experiments.message_unlock_price.variation',
'count': {'$sum': 1},
}},
],
'as': 'variation_data',
}},
{'$unwind': '$variation_data'},
{'$project': {
'_id': 1,
'purchase_amount': 1,
'paid_user_count': 1,
'total_user_count': '$variation_data.count',
}},
{'$addFields': {
'since': start,
'arpu': {'$divide': ['$purchase_amount', '$total_user_count']},
'arppu': {'$divide': ['$purchase_amount', '$paid_user_count']},
}},
{'$sort': {'_id': 1}},
]);
ref:
https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/#join-conditions-and-uncorrelated-sub-queries
or
ref:
https://stackoverflow.com/questions/37086387/multiple-join-conditions-using-the-lookup-operator
https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/#specify-multiple-join-conditions-with-lookup
Count Documents In Another Collection With $lookup
(JOIN)
ref:
https://docs.mongodb.com/manual/reference/operator/aggregation/lookup/#equality-match
Use $lookup
as findOne()
Which Returns An Object
Use $lookup
and $unwind
.
ref:
https://stackoverflow.com/questions/37691727/how-to-use-mongodbs-aggregate-lookup-as-findone
Collapse Documents In An Array
JSON output:
Do Pagination With $facet
And $project
ref:
https://docs.mongodb.com/manual/reference/operator/aggregation/facet/
https://docs.mongodb.com/manual/reference/operator/aggregation/project/
Perform $facet
+ $project
=> Unwrap with $unwind
=> Do $facet
+ $project
Again
Do $group
First To Reduce Numbers Of $lookup
Calls
ref:
https://docs.mongodb.com/manual/reference/operator/aggregation/group/
Copy Collections To Another Database
ref:
https://stackoverflow.com/questions/11554762/how-to-copy-a-collection-from-one-database-to-another-in-mongodb
Sadly, cloneCollection()
cannot clone collections from one local database to another local database.
ref:
https://docs.mongodb.com/manual/reference/command/cloneCollection/
Useful Tools
Backup
ref:
https://docs.mongodb.com/manual/reference/program/mongodump/
Restore
This kind of error typically indicates some sort of issue with data corruption, which is often caused by problems with the underlying storage device, file system or network connection.
ref:
https://docs.mongodb.com/manual/reference/program/mongorestore/
Profiling
You could also set the profiling level to 2
to record every query.
ref:
https://docs.mongodb.com/manual/tutorial/manage-the-database-profiler/
https://stackoverflow.com/questions/15204341/mongodb-logging-all-queries
ref:
https://github.com/mrsarm/mongotail
Monitoring
ref:
https://docs.mongodb.com/manual/reference/program/mongotop/
https://docs.mongodb.com/manual/reference/program/mongostat/
ref:
https://github.com/rueckstiess/mtools