Insight on MongoDB Change Stream - Abhishek.D, Mydbops Team

Insights on MongoDB change streams
Presenter by
Abhishek D
Junior Database Engineer @ Mydbops
Mydbops 14th MyWebinar
www.mydbops.com info@mydbops.com

Abhishek D
Junior MongoDB Database Engineer
abhishek@mydbops.com
linkedin.com/in/abhishek-d-512077189
About Me

Mydbops at a Glance
● Founded in 2015, HQ in Bangalore India, 70+ Employees.
● Mydbops is on Database Consulting with core specialization on MySQL, MongoDB and PostgreSQL
Administration and Support.
● Mydbops was created with a motto of developing a DevOPS model for Database Administration.
● We help organisations to scale in MySQL/Mongodb/postgresql and implement the advanced technologies
in MySQL/Mongodb/PostgreSQL.

Mydbops at a Glance
Happy Customers
4

Agenda
● Intro
● Use cases
● Features and its advantages
● Demo
● Q/A

Change Stream overview
● Change stream was Introduced in MongoDB 3.6
● Change stream enables application to stream real time data changes.

Change Stream overview
● Change stream cursors can be open for collection , database or an entire deployment
● Change stream uses the aggregation framework

Change Stream Availability
● Change streams are available for replica sets and sharded clusters only.
● Replica sets and sharded clusters must use wiredtiger storage engine.
● From MongoDB 4.2 change streams are available regardless of the “majority” read concern that is the
read concern “majority” can either be enabled or disabled.

Change Stream Availability
● In MongoDB 4.0 and earlier versions change streams are available only if the “majority” read concern is
enabled.
● Change streams can be done on all databases and collections apart from admin, config and local
databases.
● Change streams database level deployment watch is only available in MongoDB version 4.0 and above.

Use cases
Real time
analytics
Real time Notifications
IOT and data pipeline to
other systems
Event Driven Architecture

Features
● Filterble: Applications can filter changes to receive only those change notifications they need.
● Resumable :Change streams are resumable because each response comes with a resume token. Using the token,
an application can start the stream where it left off (if it ever disconnects).
● In order: Change notifications occur in the same order that the database was updated.

Features
● Secure : Only users with rights to read a collection can create a change stream on that collection.
● Durable - Change streams only include majority-committed changes. This is so every change seen by listening
applications is durable in failure scenarios, such as electing a new primary.

Advantages
Reduced development effort
Real time stream of data
changes
Reduced Infrastructural
cost.
Easy to Integrate
Higher ROI

Change Streams Custom Management
InBuilt Feature
Custom Built App
Simple to implement Complex
Quicker development Loads of development
effort
Minimal maintenance cost Higher maintenance cost

Access control
● To open a change stream against specific collection, applications must have privileges that grant changeStream and find
actions on the corresponding collection.
{ resource: { db: <dbname>, collection: <collection> }, actions: [ "find","changeStream" ] }
● To open a change stream on a single databases, applications must have privileges that grant changeStream and find
actions on all non-system collections in a database.
{ resource: { db: <dbname>, collection: "" }, actions: [ "find", "changeStream" ] }
● To open a change stream on an entire deployment, applications must have privileges that grant changeStream and find
actions on all non-system collections for all databases in the deployment.
{ resource: { db: "", collection: "" }, actions: [ "find", "changeStream" ] }

Opening a change stream
● For a replica set, issue the open change stream operation from any of the data-bearing members.
● For a sharded cluster, issue the open change stream operation from the mongos.
● The following example uses stream to process the change events.
const collection = db.collection('inventory');
const changeStream = collection.watch();
changeStream.on('change', next => {
// process next document
});

Change Events
● Change stream response document.
{
_id : { <BSON Object> },
"operationType" : "<operation>",
"fullDocument" : { <document> },
"ns" : {
"db" : "<database>",
"coll" : "<collection>"
},
"to" : {
"db" : "<database>",
"coll" : "<collection>"
},

Change Events
"documentKey" : { "_id" : <value> },
"updateDescription" : {
"updatedFields" : { <document> },
"removedFields" : [ "<field>", ... ],
"truncatedArrays" : [
{ "field" : <field>, "newSize" : <integer> },
...
]
},
"clusterTime" : <Timestamp>,
"txnNumber" : <NumberLong>,
"lsid" : {
"id" : <UUID>,
"uid" : <BinData>
}
}

Change Events
● Insert Event:
{
_id: { < Resume Token > },
operationType: 'insert',
clusterTime: <Timestamp>,
ns: {
db: 'engineering',
coll: 'users'
},
documentKey: {
userName: 'alice123',
_id: ObjectId("599af247bb69cd89961c986d")
},
fullDocument: {
_id: ObjectId("599af247bb69cd89961c986d"),
userName: 'alice123',
name: 'Alice'
}
}

Change Events
● Update event.
{
operationType: 'update',
ns: {
db: 'engineering',
coll: 'users'
},
documentKey: {
_id: ObjectId("58a4eb4a30c75625e00d2820")
},
updateDescription: {
updatedFields: {
email: 'alice@10gen.com'
},
removedFields: ['phoneNumber'],
truncatedArrays: [ {
"field" : "vacation_time",
"newSize" : 36
} ]
}
}

Change Events
● Delete event
{
operationType: 'delete',
ns: {
db: 'engineering',
coll: 'users'
},
documentKey: {
_id: ObjectId("599af247bb69cd89961c986d")
}
}

Change Events
● Drop event
{
operationType: 'drop',
ns: {
db: 'engineering',
coll: 'users'
}
}

Change Events
● Dropdatabase event
{
operationType: 'dropDatabase',
ns: {
db: 'engineering'
}
}

References
https://www.mongodb.com/basics/change-streams
https://www.mongodb.com/blog/post/whats-new-in-mongodb-36-part-1-speed-to-
develop
https://docs.mongodb.com/manual/changeStreams/
https://www.youtube.com/watch?v=hPMLuHH4ZAc&t

Insight on MongoDB Change Stream - Abhishek.D, Mydbops Team

Insight on MongoDB Change Stream - Abhishek.D, Mydbops Team

More Related Content

What's hot

Similar to Insight on MongoDB Change Stream - Abhishek.D, Mydbops Team

More from Mydbops

Recently uploaded

In this document

Insight on MongoDB Change Stream - Abhishek.D, Mydbops Team