Beyond the Basics 2: Aggregation Framework
Download as pptx, pdf0 likes2,052 views
The aggregation framework is one of the most powerful analytical tools available with MongoDB. Learn how to create a pipeline of operations that can reshape and transform your data and apply a range of analytics functions and calculations to produce summary results across a data set.
![11
Example Document
MongoDB Enterprise > db.members.find( { "batchID" : 138,
"member.member_name" : "Joe Drumgoole" },
{ "_id" : 0, "member.chapters" : 0 } ).pretty()
{
"member" : {
"city" : "Dublin",
"events_attended" : 19,
"last_access_time" : ISODate("2017-04-25T12:40:55Z"),
"country" : "Ireland",
"member_id" : 99473492,
"is_organizer" : true,
"photo_thumb_url" : "https://secure.meetupstatic.com/photos/member/e/5/0/1/thumb_255178625.jpeg",
"location" : {
"type" : "Point",
"coordinates" : [
-6.25,
53.33000183105469
]
},
"member_name" : "Joe Drumgoole",
"join_time" : ISODate("2013-10-30T17:05:31Z")
},
"timestamp" : ISODate("2017-04-26T10:13:54.079Z"),
"batchID" : 138
}](https://image.slidesharecdn.com/02btbanalytics-1-170427141842/85/Beyond-the-Basics-2-Aggregation-Framework-11-320.jpg)
![14
Let’s Query for Groups
JD10Gen:apps jdrumgoole$ ./mug_analytics_main.py --stats groups --url mongodb-belgium DublinMUG
Processing : ['mongodb-belgium', 'DublinMUG']
pro
db.groups.aggregate( [
{"$match": {"batchID": 138, "group.urlname": {"$in": ["mongodb-belgium", "DublinMUG"]}}},
{"$project": {"founded": "$group.founded_date", "urlname": "$group.urlname", "_id": 0, "members":
"$group.member_count"}},
])
{'founded': '11-Sep-2013 14:05', 'members': 399, 'urlname': u'mongodb-belgium'}
{'founded': '14-Mar-2012 17:40', 'members': 847, 'urlname': u'DublinMUG'}
Wrote 2 records
JD10Gen:apps jdrumgoole$](https://image.slidesharecdn.com/02btbanalytics-1-170427141842/85/Beyond-the-Basics-2-Aggregation-Framework-14-320.jpg)
![16
Search for New Members
JD10Gen:apps jdrumgoole$ ./mug_analytics_main.py --stats newmembers --url DublinMUG --sort join_date --format
csv --direction ascending --limit 10
Processing : ['DublinMUG']
Sorting on 'join_date' direction = 'ascending'
db.members.aggregate([
{"$match": {"batchID": 138}},
{"$unwind": "$member.chapters"},
{"$match": {"member.chapters.urlname": {"$in": ["DublinMUG"]}}},
{"$project": {"join_date": "$member.join_time", "_id": 0, "group": "$member.chapters.urlname", "name":
"$member.member_name"}},
{"$limit": 10}])
group,name,join_date
DublinMUG,Gosia,17-Apr-2017 12:51
DublinMUG,Luke Shiels,15-Apr-2017 14:00
DublinMUG,Silvia Sirbu,11-Apr-2017 12:00
DublinMUG,Steeve P.,04-Apr-2017 09:47
DublinMUG,Dafei W,30-Mar-2017 11:36
DublinMUG,Ross Norman,13-Mar-2017 11:30
DublinMUG,Grzegorz F.,08-Mar-2017 10:25
DublinMUG,Lucas Sacramento,07-Mar-2017 11:05
DublinMUG,David Blount,06-Mar-2017 12:33
DublinMUG,Luca Ballerini,06-Mar-2017 10:41
Wrote 10 records
JD10Gen:apps jdrumgoole$](https://image.slidesharecdn.com/02btbanalytics-1-170427141842/85/Beyond-the-Basics-2-Aggregation-Framework-16-320.jpg)
![17
Search for New Members This Year
JD10Gen:apps jdrumgoole$ ./mug_analytics_main.py --stats newmembers --url DublinMUG --sort join_date --format csv --
direction ascending --limit 10 --start 1-Jan-2017
Processing : ['DublinMUG']
Sorting on 'join_date' direction = 'ascending'
db.members.aggregate([
{"$match": {"batchID": 138}},
{"$unwind": "$member.chapters"},
{"$match": {"member.chapters.urlname": {"$in": ["DublinMUG"]}}},
{"$match": {"member.join_time": {"$gte": "2017-01-01T00:00:00"}}},
{"$project": {"join_date": "$member.join_time", "_id": 0, "group": "$member.chapters.urlname", "name":
"$member.member_name"}},
{"$limit": 10}])
group,name,join_date
DublinMUG,Gosia,17-Apr-2017 12:51
DublinMUG,Luke Shiels,15-Apr-2017 14:00
DublinMUG,Silvia Sirbu,11-Apr-2017 12:00
DublinMUG,Steeve P.,04-Apr-2017 09:47
DublinMUG,Dafei W,30-Mar-2017 11:36
DublinMUG,Ross Norman,13-Mar-2017 11:30
DublinMUG,Grzegorz F.,08-Mar-2017 10:25
DublinMUG,Lucas Sacramento,07-Mar-2017 11:05
DublinMUG,David Blount,06-Mar-2017 12:33
DublinMUG,Luca Ballerini,06-Mar-2017 10:41
Wrote 10 records](https://image.slidesharecdn.com/02btbanalytics-1-170427141842/85/Beyond-the-Basics-2-Aggregation-Framework-17-320.jpg)
![18
Turn an Aggregation into a View
• Only supported on MongoDB 3.4
• Views are a non-materialised view on a collection
MongoDB Enterprise > db.createView( "batch138",
"members",
[ { "$match" : { "batchID" : 138 }} ] )
{ "ok" : 1 }
MongoDB Enterprise >
• A view persists and will return new results each time a find is run
• A view looks just like a collection
• Must turn 3.4 compatibility on
MongoDB Enterprise > db.adminCommand( { setFeatureCompatibilityVersion: "3.4"} )](https://image.slidesharecdn.com/02btbanalytics-1-170427141842/85/Beyond-the-Basics-2-Aggregation-Framework-18-320.jpg)
More Related Content
What's hot (20)
Similar to Beyond the Basics 2: Aggregation Framework (20)
More from MongoDB (20)
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=560&fit=bounds)
Recently uploaded (20)
![[Eddie Lee] Capstone Project - AI PM Bootcamp - DataFox.pdf](https://cdn.slidesharecdn.com/ss_thumbnails/eddieleecapstoneproject-aipmbootcamp-datafox-250607001838-bf03fa6b-thumbnail.jpg?width=560&fit=bounds)
Beyond the Basics 2: Aggregation Framework
- 2. Beyond The Basics : Part 2 Analytics and the Aggregation Framework Joe Drumgoole Director of Developer Advocacy, EMEA @jdrumgoole V1.1
- 3. 3 Beyond The Basics – Storage Engines • What storage engines are and how to pick them – Aggregation Framework • How to deploy advanced analytics processing right inside the database – The BI Connector • How to create visualisations and dashboards from your MongoDB data – Authentication and Authorisation • How to secure MongoDB, both on-premise and in the cloud
- 4. 4 The Aggregation Framework • An analytics engine for MongoDB • What is analytics? • Think of the two types of database, OLTP, OLAP • OLTP : Online Transaction Processing – Airline booking, – ATMs, – Taxi booking • OLAP : Online Analytical Processing – Which tickets make us most money? – When do we need to refill our ATMs? – How many cabs do we need to service the West End of London?
- 5. 5 OLTP – Online Transaction processing
- 6. 6 OLTP – Online Transaction processing sum() avg()
- 7. 7 The Aggregation Framework – A Processing Pipeline Match Project Group SortLimit • Think unix pipeline • The output of one stage is passed to the input of the next stage • Each stage performs one job • Stages can be repeated • Output is a cursor, a new collection or a view
- 8. 8 Typical Goals of Aggregation Framework • Columnar Analytics • Reshaping data • Unwinding arrays into individual documents • Linking collections together • Generating new data from old (collections and views)
- 9. 9 Pipeline Operators • $match Filter documents • $project Reshape documents • $group Summarize documents • $out Create new collections • $sort Order documents • $limit/$skip Paginate documents • $lookup Join two collections together • $unwind Expand an array
- 10. 10 Example Pipeline Match Project Group Sort Out • Find content • Standard query • Uses indexes • Reduce doc count • Use first • Select content • Remove fields • Add fields • Reduce doc size • Looks at every doc • Collect content • Sum, Avg etc. • Rewrite _id • Reduce doc count • Looks at every doc • Sort on fields • Several sorts allowed • Ascending or descending • 100mb limit • Allow Disk Use • New collection • $out overwrites • Only one per aggregate • Last member
- 11. 11 Example Document MongoDB Enterprise > db.members.find( { "batchID" : 138, "member.member_name" : "Joe Drumgoole" }, { "_id" : 0, "member.chapters" : 0 } ).pretty() { "member" : { "city" : "Dublin", "events_attended" : 19, "last_access_time" : ISODate("2017-04-25T12:40:55Z"), "country" : "Ireland", "member_id" : 99473492, "is_organizer" : true, "photo_thumb_url" : "https://secure.meetupstatic.com/photos/member/e/5/0/1/thumb_255178625.jpeg", "location" : { "type" : "Point", "coordinates" : [ -6.25, 53.33000183105469 ] }, "member_name" : "Joe Drumgoole", "join_time" : ISODate("2013-10-30T17:05:31Z") }, "timestamp" : ISODate("2017-04-26T10:13:54.079Z"), "batchID" : 138 }
- 12. 12 Meetup Data
- 13. 13 Group Document MongoDB Enterprise > db.groups.findOne({ "batchID": 138 }, { "group.photos" : 0, "group.topics" : 0, "group.location" : 0, "group.description" : 0, "group.organizers" : 0, "group.category" : 0, "_id" : 0 } ) { "batchID" : 138, "timestamp" : ISODate("2017-04-26T10:12:06.388Z"), "group" : { "rsvps_per_event" : 39.285701751708984, "repeat_rsvpers" : 62, "upcoming_events" : 0, "gender_female" : 0.039500001817941666, "pro_join_date" : ISODate("2017-04-10T18:11:49Z"), "id" : 10209022, "city" : "Gent", "member_count" : 399, "average_age" : 35.15570068359375, "status" : "Active", "founded_date" : ISODate("2013-09-11T14:05:30Z"), "urlname" : "mongodb-belgium", "gender_male" : 0.9473999738693237, "name" : "MongoDB Belgium", "last_event" : ISODate("2015-06-16T18:00:00Z"), "country" : "Belgium", "gender_unknown" : 0.013199999928474426, "past_events" : 7, "gender_other" : 0, "past_rsvps" : 275 } }
- 14. 14 Let’s Query for Groups JD10Gen:apps jdrumgoole$ ./mug_analytics_main.py --stats groups --url mongodb-belgium DublinMUG Processing : ['mongodb-belgium', 'DublinMUG'] pro db.groups.aggregate( [ {"$match": {"batchID": 138, "group.urlname": {"$in": ["mongodb-belgium", "DublinMUG"]}}}, {"$project": {"founded": "$group.founded_date", "urlname": "$group.urlname", "_id": 0, "members": "$group.member_count"}}, ]) {'founded': '11-Sep-2013 14:05', 'members': 399, 'urlname': u'mongodb-belgium'} {'founded': '14-Mar-2012 17:40', 'members': 847, 'urlname': u'DublinMUG'} Wrote 2 records JD10Gen:apps jdrumgoole$
- 15. 15 Attendee Doc MongoDB Enterprise > db.attendees.findOne( { "batchID" : 138 }, { "info.event.description" : 0, "_id" : 0, "info.event.group" : 0, "info.event.venue" : 0, "info.event.rating" : 0 } ) { "info" : { "attendee" : { "status" : "attended", "member" : { "name" : "Former member" }, "rsvp" : { "response" : "yes", } }, "event" : { "status" : "past", "event_url" : "https://www.meetup.com/mongodb-belgium/events/162104572/", "created" : ISODate("2014-01-23T09:10:20Z"), "rsvp_limit" : 75, "updated" : ISODate("2014-03-20T11:49:00Z"), "visibility" : "public", "yes_rsvp_count" : 75, "time" : ISODate("2014-03-18T18:00:00Z"), "headcount" : 0, "id" : "162104572", "name" : "MongoDB Belgium #1: the kickoff" } }, "timestamp" : ISODate("2017-04-26T10:14:30.129Z"), "batchID" : 138 }
- 16. 16 Search for New Members JD10Gen:apps jdrumgoole$ ./mug_analytics_main.py --stats newmembers --url DublinMUG --sort join_date --format csv --direction ascending --limit 10 Processing : ['DublinMUG'] Sorting on 'join_date' direction = 'ascending' db.members.aggregate([ {"$match": {"batchID": 138}}, {"$unwind": "$member.chapters"}, {"$match": {"member.chapters.urlname": {"$in": ["DublinMUG"]}}}, {"$project": {"join_date": "$member.join_time", "_id": 0, "group": "$member.chapters.urlname", "name": "$member.member_name"}}, {"$limit": 10}]) group,name,join_date DublinMUG,Gosia,17-Apr-2017 12:51 DublinMUG,Luke Shiels,15-Apr-2017 14:00 DublinMUG,Silvia Sirbu,11-Apr-2017 12:00 DublinMUG,Steeve P.,04-Apr-2017 09:47 DublinMUG,Dafei W,30-Mar-2017 11:36 DublinMUG,Ross Norman,13-Mar-2017 11:30 DublinMUG,Grzegorz F.,08-Mar-2017 10:25 DublinMUG,Lucas Sacramento,07-Mar-2017 11:05 DublinMUG,David Blount,06-Mar-2017 12:33 DublinMUG,Luca Ballerini,06-Mar-2017 10:41 Wrote 10 records JD10Gen:apps jdrumgoole$
- 17. 17 Search for New Members This Year JD10Gen:apps jdrumgoole$ ./mug_analytics_main.py --stats newmembers --url DublinMUG --sort join_date --format csv -- direction ascending --limit 10 --start 1-Jan-2017 Processing : ['DublinMUG'] Sorting on 'join_date' direction = 'ascending' db.members.aggregate([ {"$match": {"batchID": 138}}, {"$unwind": "$member.chapters"}, {"$match": {"member.chapters.urlname": {"$in": ["DublinMUG"]}}}, {"$match": {"member.join_time": {"$gte": "2017-01-01T00:00:00"}}}, {"$project": {"join_date": "$member.join_time", "_id": 0, "group": "$member.chapters.urlname", "name": "$member.member_name"}}, {"$limit": 10}]) group,name,join_date DublinMUG,Gosia,17-Apr-2017 12:51 DublinMUG,Luke Shiels,15-Apr-2017 14:00 DublinMUG,Silvia Sirbu,11-Apr-2017 12:00 DublinMUG,Steeve P.,04-Apr-2017 09:47 DublinMUG,Dafei W,30-Mar-2017 11:36 DublinMUG,Ross Norman,13-Mar-2017 11:30 DublinMUG,Grzegorz F.,08-Mar-2017 10:25 DublinMUG,Lucas Sacramento,07-Mar-2017 11:05 DublinMUG,David Blount,06-Mar-2017 12:33 DublinMUG,Luca Ballerini,06-Mar-2017 10:41 Wrote 10 records
- 18. 18 Turn an Aggregation into a View • Only supported on MongoDB 3.4 • Views are a non-materialised view on a collection MongoDB Enterprise > db.createView( "batch138", "members", [ { "$match" : { "batchID" : 138 }} ] ) { "ok" : 1 } MongoDB Enterprise > • A view persists and will return new results each time a find is run • A view looks just like a collection • Must turn 3.4 compatibility on MongoDB Enterprise > db.adminCommand( { setFeatureCompatibilityVersion: "3.4"} )
- 19. 19 Useful Links • The Aggregation Python class https://github.com/jdrumgoole/mongodb_utils/blob/master/mongodb_utils/agg.py • Aggregation docs https://docs.mongodb.com/manual/aggregation/ • MongoDB Views in 3.4 https://docs.mongodb.com/manual/core/views/
- 20. 20 Q&A