SlideShare a Scribd company logo
Aggregation Framework
in MongoDB
Overview - Part-1
What is Aggregation
English Definition:- The act of gathering
something together.
Database Definition:- Aggregation are
operations that process on the data sets,
group some data records, do some
computation on that records and return
computed results.
Aggregration Approach in
MongoDB
● Aggregation Pipeline
● Map-Reduce
● Single Purpose Aggregation Operations
● Hadoop Connector
Aggregation-Pipeline
Similar to pipeline in UNIX.
In Unix -
Cat file.txt | grep abc | wc -l
In MongoDB -
Group Limit SortCollection Output
MapReduce
● Two Phase process – Mapper & Reducer.
● Use JavaScript function for map-reduce
job.
● Less efficient as it run over only one thread.
● Finalise stage to make final modification.
Aggregation Framework in MongoDB Overview Part-1
Aggregarion Vs. MapReduce
Easy to use, just need to use the build
in operators.
Complex and steep learning curve.
Supports non-sharded and sharded
input collections.
Supports non-sharded and sharded
input collections.
Returns results inline. Return result in inline, new collection,
merge, replace, reduce.
Limited to the operators and
expressions supported by the
aggregation pipeline.
Custom map, reduce and finalize
JavaScript functions offer flexibility to
aggregation logic.
Single Purpose Aggregation
Operation
● Applicable for single purpose of
aggregation, like count, distinct, group.
● Limited scope as compared to aggregation
pipeline and map-reduce.
● group does not support data in sharded
collections, and result of operation should
not be more than 16 MB.
Aggregation Pipeline Operators
● $group
● $match
● $project
● $limit
● $sort
● $unwind
● $skip
SQL to Aggregation Mapping Chart
SQL Operators Aggregation Operators
WHERE $match
GROUP BY $group
SELECT $project
LIMIT $limit
ORDER BY $sort
SUM() $sum
COUNT() $sum
JOIN $unwind (Not exact operator as JOIN
works, but unwinds the array
embedded in the document)
Examples
SQL Query Aggregation Query
SELECT COUNT(*) AS COUNT
FROM EMPLOYEE
db.employee.aggregate([
{$group: {_id:null, count: { $sum:1} } }
])
Explaination :
Group by : on nothing
Sum : Just add 1 to count field for each
record
SELECT SUM(SALARY) AS TOTAL
FROM EMPLOYEE
db.employee.aggregate([
{$group: {_id:null, total:
{ $sum:”$salary”} } }
])
Explaination :
Group by : on nothing
Sum: do the sum of value of salary
field of each doc and put result in total.
Example Cont.
SELECT DEPARTMENT_ID,
SUM(SALARY) AS TOTAL FROM
EMPLOYEE GROUP BY
DEPARTMENT_ID ORDER BY
TOTAL
db.employee.aggregate( [
{ $group: { _id: “$department_id”, total:
{ $sum:”$salary” } },
{ $sort: { total : 1 } } }
] )
Explanation:
Group by : department,
Sum : salary,
Order by : total of salary for each
department
Example Cont.
SELECT DEPT_ID, SUM(SALARY)
AS TOTAL FROM EMPLOYEE
WHERE AGE>25 GROUB BY
DEPT_ID, HAVING TOTAL > 5000
db.employee.aggregate( [
{ $match : { age : {$gt: 25 } } }
{ $group: { _id: “$dept_id”, total:
{ $sum: ”$salary” } } },
{ $match: { total: {$gt:5000 } } }
] )
Explanation:
Group by : department id,
Sum : salary,
Having: on total of each department
salary
Example Cont.
SELECT DEPT_ID, SUM(SALARY) AS
TOTAL FROM EMPLOYEE WHERE
AGE>25 GROUB BY DEPT_ID,
HAVING TOTAL > 5000
db.employee.aggregate( [
{ $match : { age : {$gt: 25 } } }
{ $group: { _id: “$dept_id”, total:
{ $sum: ”$salary” } } },
{ $match: { total: {$gt:5000 } } }
] )
Explanation:
Group by : department id,
Sum : salary,
Having: on total of each department
salary
$unwind Operator
Decompose the embedded array into flat
document and relate each entry in the
array with outer fields.
Example :-
{ _id: “blog”, tags: [ “social”, “economic” ] }
{ _id: “blog”, { _id: “blog”
tags: “social” } tags: “economic” }
Mongo Aggregation Optimization
MongoDB re-arranged the pipeline
operations to optimize the aggregation
performance.
● Pipeline Sequence Optimization.
● Projection Optimization.
Mongo Aggregation Optimization
MongoDB re-arranged the pipeline
operations to optimize the aggregation
performance.
● Pipeline Sequence Optimization.
● Projection Optimization.
Pipeline Sequence Optimization
● $sort + $skip + $limit
{ $sort: { salary : -1 } }, { $sort: { salary: -1 } },
{ $skip: 10 }, { $limit: 20 },
{ $limit: 10 } { $skip: 10}
Projection Optimization
● $project
● Reduce the amount of data passing
through channels of operation and will help
in performace improvement.
● Below example will only emit salary.
db.employee.aggregate(
[ {$match: {“name”: “xyz” } }
{ $project: { salary:1, _id:0} } ]
)
Restriction
● Output BSON document cannot exceed the
16 MB of data. If exceed will throw error.
● If single aggregation operation consumes
more than 10 percent of system RAM, the
operation will produce error.
Aggregation Examples
● Download data from the below link:
http://media.mongdb.org/zips.json
● Practice sample problem from the below
links:
http://docs.mongodb.org/manual/tutorial/aggregatio-zip-code-data-set/
References
● http://docs.mongodb.org/
Good Examples:-
● http://rubayeet.wordpress.com/2013/12/29/web-analytics
-using-mongodb-aggregation-framework/
● http://derickrethans.nl/aggregation-framework.html
● http://architects.dzone.com/articles/using-mongodb
-aggregation
Anuj Jain
● ajain@equalexperts.com
● anuj2jain@gmail.com

More Related Content

What's hot (20)

MongoDB Aggregation
MongoDB Aggregation MongoDB Aggregation
MongoDB Aggregation
Amit Ghosh
 
MongoDB - Aggregation Pipeline
MongoDB - Aggregation PipelineMongoDB - Aggregation Pipeline
MongoDB - Aggregation Pipeline
Jason Terpko
 
Data Processing and Aggregation with MongoDB
Data Processing and Aggregation with MongoDB Data Processing and Aggregation with MongoDB
Data Processing and Aggregation with MongoDB
MongoDB
 
Agg framework selectgroup feb2015 v2
Agg framework selectgroup feb2015 v2Agg framework selectgroup feb2015 v2
Agg framework selectgroup feb2015 v2
MongoDB
 
Webinar: Exploring the Aggregation Framework
Webinar: Exploring the Aggregation FrameworkWebinar: Exploring the Aggregation Framework
Webinar: Exploring the Aggregation Framework
MongoDB
 
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB
 
Mongodb Aggregation Pipeline
Mongodb Aggregation PipelineMongodb Aggregation Pipeline
Mongodb Aggregation Pipeline
zahid-mian
 
Webinar: Data Processing and Aggregation Options
Webinar: Data Processing and Aggregation OptionsWebinar: Data Processing and Aggregation Options
Webinar: Data Processing and Aggregation Options
MongoDB
 
Analytics with MongoDB Aggregation Framework and Hadoop Connector
Analytics with MongoDB Aggregation Framework and Hadoop ConnectorAnalytics with MongoDB Aggregation Framework and Hadoop Connector
Analytics with MongoDB Aggregation Framework and Hadoop Connector
Henrik Ingo
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
Nosh Petigara
 
Beyond the Basics 2: Aggregation Framework
Beyond the Basics 2: Aggregation Framework Beyond the Basics 2: Aggregation Framework
Beyond the Basics 2: Aggregation Framework
MongoDB
 
Webinar: Working with Graph Data in MongoDB
Webinar: Working with Graph Data in MongoDBWebinar: Working with Graph Data in MongoDB
Webinar: Working with Graph Data in MongoDB
MongoDB
 
Getting Started with MongoDB and NodeJS
Getting Started with MongoDB and NodeJSGetting Started with MongoDB and NodeJS
Getting Started with MongoDB and NodeJS
MongoDB
 
Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014
MongoDB
 
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB
 
Latinoware
LatinowareLatinoware
Latinoware
kchodorow
 
"Powerful Analysis with the Aggregation Pipeline (Tutorial)"
"Powerful Analysis with the Aggregation Pipeline (Tutorial)""Powerful Analysis with the Aggregation Pipeline (Tutorial)"
"Powerful Analysis with the Aggregation Pipeline (Tutorial)"
MongoDB
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
antoinegirbal
 
Webinarserie: Einführung in MongoDB: “Back to Basics” - Teil 3 - Interaktion ...
Webinarserie: Einführung in MongoDB: “Back to Basics” - Teil 3 - Interaktion ...Webinarserie: Einführung in MongoDB: “Back to Basics” - Teil 3 - Interaktion ...
Webinarserie: Einführung in MongoDB: “Back to Basics” - Teil 3 - Interaktion ...
MongoDB
 
Joins and Other Aggregation Enhancements Coming in MongoDB 3.2
Joins and Other Aggregation Enhancements Coming in MongoDB 3.2Joins and Other Aggregation Enhancements Coming in MongoDB 3.2
Joins and Other Aggregation Enhancements Coming in MongoDB 3.2
MongoDB
 
MongoDB Aggregation
MongoDB Aggregation MongoDB Aggregation
MongoDB Aggregation
Amit Ghosh
 
MongoDB - Aggregation Pipeline
MongoDB - Aggregation PipelineMongoDB - Aggregation Pipeline
MongoDB - Aggregation Pipeline
Jason Terpko
 
Data Processing and Aggregation with MongoDB
Data Processing and Aggregation with MongoDB Data Processing and Aggregation with MongoDB
Data Processing and Aggregation with MongoDB
MongoDB
 
Agg framework selectgroup feb2015 v2
Agg framework selectgroup feb2015 v2Agg framework selectgroup feb2015 v2
Agg framework selectgroup feb2015 v2
MongoDB
 
Webinar: Exploring the Aggregation Framework
Webinar: Exploring the Aggregation FrameworkWebinar: Exploring the Aggregation Framework
Webinar: Exploring the Aggregation Framework
MongoDB
 
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB
 
Mongodb Aggregation Pipeline
Mongodb Aggregation PipelineMongodb Aggregation Pipeline
Mongodb Aggregation Pipeline
zahid-mian
 
Webinar: Data Processing and Aggregation Options
Webinar: Data Processing and Aggregation OptionsWebinar: Data Processing and Aggregation Options
Webinar: Data Processing and Aggregation Options
MongoDB
 
Analytics with MongoDB Aggregation Framework and Hadoop Connector
Analytics with MongoDB Aggregation Framework and Hadoop ConnectorAnalytics with MongoDB Aggregation Framework and Hadoop Connector
Analytics with MongoDB Aggregation Framework and Hadoop Connector
Henrik Ingo
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
Nosh Petigara
 
Beyond the Basics 2: Aggregation Framework
Beyond the Basics 2: Aggregation Framework Beyond the Basics 2: Aggregation Framework
Beyond the Basics 2: Aggregation Framework
MongoDB
 
Webinar: Working with Graph Data in MongoDB
Webinar: Working with Graph Data in MongoDBWebinar: Working with Graph Data in MongoDB
Webinar: Working with Graph Data in MongoDB
MongoDB
 
Getting Started with MongoDB and NodeJS
Getting Started with MongoDB and NodeJSGetting Started with MongoDB and NodeJS
Getting Started with MongoDB and NodeJS
MongoDB
 
Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014Hadoop - MongoDB Webinar June 2014
Hadoop - MongoDB Webinar June 2014
MongoDB
 
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB
 
"Powerful Analysis with the Aggregation Pipeline (Tutorial)"
"Powerful Analysis with the Aggregation Pipeline (Tutorial)""Powerful Analysis with the Aggregation Pipeline (Tutorial)"
"Powerful Analysis with the Aggregation Pipeline (Tutorial)"
MongoDB
 
Introduction to MongoDB
Introduction to MongoDBIntroduction to MongoDB
Introduction to MongoDB
antoinegirbal
 
Webinarserie: Einführung in MongoDB: “Back to Basics” - Teil 3 - Interaktion ...
Webinarserie: Einführung in MongoDB: “Back to Basics” - Teil 3 - Interaktion ...Webinarserie: Einführung in MongoDB: “Back to Basics” - Teil 3 - Interaktion ...
Webinarserie: Einführung in MongoDB: “Back to Basics” - Teil 3 - Interaktion ...
MongoDB
 
Joins and Other Aggregation Enhancements Coming in MongoDB 3.2
Joins and Other Aggregation Enhancements Coming in MongoDB 3.2Joins and Other Aggregation Enhancements Coming in MongoDB 3.2
Joins and Other Aggregation Enhancements Coming in MongoDB 3.2
MongoDB
 

Similar to Aggregation Framework in MongoDB Overview Part-1 (20)

Experiment no 05
Experiment no 05Experiment no 05
Experiment no 05
Ankit Dubey
 
SH 2 - SES 3 - MongoDB Aggregation Framework.pptx
SH 2 - SES 3 -  MongoDB Aggregation Framework.pptxSH 2 - SES 3 -  MongoDB Aggregation Framework.pptx
SH 2 - SES 3 - MongoDB Aggregation Framework.pptx
MongoDB
 
Joins and Other MongoDB 3.2 Aggregation Enhancements
Joins and Other MongoDB 3.2 Aggregation EnhancementsJoins and Other MongoDB 3.2 Aggregation Enhancements
Joins and Other MongoDB 3.2 Aggregation Enhancements
Andrew Morgan
 
MongoDB Aggregations Indexing and Profiling
MongoDB Aggregations Indexing and ProfilingMongoDB Aggregations Indexing and Profiling
MongoDB Aggregations Indexing and Profiling
Manish Kapoor
 
Precog & MongoDB User Group: Skyrocket Your Analytics
Precog & MongoDB User Group: Skyrocket Your Analytics Precog & MongoDB User Group: Skyrocket Your Analytics
Precog & MongoDB User Group: Skyrocket Your Analytics
MongoDB
 
Learning MongoDB Aggregations in 10 Minutes
Learning MongoDB Aggregations in 10 MinutesLearning MongoDB Aggregations in 10 Minutes
Learning MongoDB Aggregations in 10 Minutes
techprane
 
MongoDB_ppt.pptx
MongoDB_ppt.pptxMongoDB_ppt.pptx
MongoDB_ppt.pptx
1AP18CS037ShirishKul
 
Mongo db aggregation guide
Mongo db aggregation guideMongo db aggregation guide
Mongo db aggregation guide
Deysi Gmarra
 
Mongo db aggregation-guide
Mongo db aggregation-guideMongo db aggregation-guide
Mongo db aggregation-guide
Dan Llimpe
 
MongoDB's New Aggregation framework
MongoDB's New Aggregation frameworkMongoDB's New Aggregation framework
MongoDB's New Aggregation framework
Chris Westin
 
Querying mongo db
Querying mongo dbQuerying mongo db
Querying mongo db
Bogdan Sabău
 
mongodb-aggregation-may-2012
mongodb-aggregation-may-2012mongodb-aggregation-may-2012
mongodb-aggregation-may-2012
Chris Westin
 
2014 bigdatacamp asya_kamsky
2014 bigdatacamp asya_kamsky2014 bigdatacamp asya_kamsky
2014 bigdatacamp asya_kamsky
Data Con LA
 
MongoDb and NoSQL
MongoDb and NoSQLMongoDb and NoSQL
MongoDb and NoSQL
TO THE NEW | Technology
 
MongoDB Aggregation MongoSF May 2011
MongoDB Aggregation MongoSF May 2011MongoDB Aggregation MongoSF May 2011
MongoDB Aggregation MongoSF May 2011
Chris Westin
 
Using MongoDB and Python
Using MongoDB and PythonUsing MongoDB and Python
Using MongoDB and Python
Mike Bright
 
2016 feb-23 pyugre-py_mongo
2016 feb-23 pyugre-py_mongo2016 feb-23 pyugre-py_mongo
2016 feb-23 pyugre-py_mongo
Michael Bright
 
MongoDB Distilled
MongoDB DistilledMongoDB Distilled
MongoDB Distilled
b0ris_1
 
Indexing and Query Performance in MongoDB.pdf
Indexing and Query Performance in MongoDB.pdfIndexing and Query Performance in MongoDB.pdf
Indexing and Query Performance in MongoDB.pdf
Malak Abu Hammad
 
Webinar: Managing Real Time Risk Analytics with MongoDB
Webinar: Managing Real Time Risk Analytics with MongoDB Webinar: Managing Real Time Risk Analytics with MongoDB
Webinar: Managing Real Time Risk Analytics with MongoDB
MongoDB
 
Experiment no 05
Experiment no 05Experiment no 05
Experiment no 05
Ankit Dubey
 
SH 2 - SES 3 - MongoDB Aggregation Framework.pptx
SH 2 - SES 3 -  MongoDB Aggregation Framework.pptxSH 2 - SES 3 -  MongoDB Aggregation Framework.pptx
SH 2 - SES 3 - MongoDB Aggregation Framework.pptx
MongoDB
 
Joins and Other MongoDB 3.2 Aggregation Enhancements
Joins and Other MongoDB 3.2 Aggregation EnhancementsJoins and Other MongoDB 3.2 Aggregation Enhancements
Joins and Other MongoDB 3.2 Aggregation Enhancements
Andrew Morgan
 
MongoDB Aggregations Indexing and Profiling
MongoDB Aggregations Indexing and ProfilingMongoDB Aggregations Indexing and Profiling
MongoDB Aggregations Indexing and Profiling
Manish Kapoor
 
Precog & MongoDB User Group: Skyrocket Your Analytics
Precog & MongoDB User Group: Skyrocket Your Analytics Precog & MongoDB User Group: Skyrocket Your Analytics
Precog & MongoDB User Group: Skyrocket Your Analytics
MongoDB
 
Learning MongoDB Aggregations in 10 Minutes
Learning MongoDB Aggregations in 10 MinutesLearning MongoDB Aggregations in 10 Minutes
Learning MongoDB Aggregations in 10 Minutes
techprane
 
Mongo db aggregation guide
Mongo db aggregation guideMongo db aggregation guide
Mongo db aggregation guide
Deysi Gmarra
 
Mongo db aggregation-guide
Mongo db aggregation-guideMongo db aggregation-guide
Mongo db aggregation-guide
Dan Llimpe
 
MongoDB's New Aggregation framework
MongoDB's New Aggregation frameworkMongoDB's New Aggregation framework
MongoDB's New Aggregation framework
Chris Westin
 
mongodb-aggregation-may-2012
mongodb-aggregation-may-2012mongodb-aggregation-may-2012
mongodb-aggregation-may-2012
Chris Westin
 
2014 bigdatacamp asya_kamsky
2014 bigdatacamp asya_kamsky2014 bigdatacamp asya_kamsky
2014 bigdatacamp asya_kamsky
Data Con LA
 
MongoDB Aggregation MongoSF May 2011
MongoDB Aggregation MongoSF May 2011MongoDB Aggregation MongoSF May 2011
MongoDB Aggregation MongoSF May 2011
Chris Westin
 
Using MongoDB and Python
Using MongoDB and PythonUsing MongoDB and Python
Using MongoDB and Python
Mike Bright
 
2016 feb-23 pyugre-py_mongo
2016 feb-23 pyugre-py_mongo2016 feb-23 pyugre-py_mongo
2016 feb-23 pyugre-py_mongo
Michael Bright
 
MongoDB Distilled
MongoDB DistilledMongoDB Distilled
MongoDB Distilled
b0ris_1
 
Indexing and Query Performance in MongoDB.pdf
Indexing and Query Performance in MongoDB.pdfIndexing and Query Performance in MongoDB.pdf
Indexing and Query Performance in MongoDB.pdf
Malak Abu Hammad
 
Webinar: Managing Real Time Risk Analytics with MongoDB
Webinar: Managing Real Time Risk Analytics with MongoDB Webinar: Managing Real Time Risk Analytics with MongoDB
Webinar: Managing Real Time Risk Analytics with MongoDB
MongoDB
 
Ad

Recently uploaded (20)

TimeSeries Machine Learning - PyData London 2025
TimeSeries Machine Learning - PyData London 2025TimeSeries Machine Learning - PyData London 2025
TimeSeries Machine Learning - PyData London 2025
Suyash Joshi
 
Developing Schemas with FME and Excel - Peak of Data & AI 2025
Developing Schemas with FME and Excel - Peak of Data & AI 2025Developing Schemas with FME and Excel - Peak of Data & AI 2025
Developing Schemas with FME and Excel - Peak of Data & AI 2025
Safe Software
 
Improving Developer Productivity With DORA, SPACE, and DevEx
Improving Developer Productivity With DORA, SPACE, and DevExImproving Developer Productivity With DORA, SPACE, and DevEx
Improving Developer Productivity With DORA, SPACE, and DevEx
Justin Reock
 
Jira Administration Training – Day 1 : Introduction
Jira Administration Training – Day 1 : IntroductionJira Administration Training – Day 1 : Introduction
Jira Administration Training – Day 1 : Introduction
Ravi Teja
 
What is Oracle EPM A Guide to Oracle EPM Cloud Everything You Need to Know
What is Oracle EPM A Guide to Oracle EPM Cloud Everything You Need to KnowWhat is Oracle EPM A Guide to Oracle EPM Cloud Everything You Need to Know
What is Oracle EPM A Guide to Oracle EPM Cloud Everything You Need to Know
SMACT Works
 
GIS and FME: The Foundation to Improve the Locate Process of Utilities
GIS and FME: The Foundation to Improve the Locate Process of UtilitiesGIS and FME: The Foundation to Improve the Locate Process of Utilities
GIS and FME: The Foundation to Improve the Locate Process of Utilities
Safe Software
 
Agentic AI: Beyond the Buzz- LangGraph Studio V2
Agentic AI: Beyond the Buzz- LangGraph Studio V2Agentic AI: Beyond the Buzz- LangGraph Studio V2
Agentic AI: Beyond the Buzz- LangGraph Studio V2
Shashikant Jagtap
 
Integration of Utility Data into 3D BIM Models Using a 3D Solids Modeling Wor...
Integration of Utility Data into 3D BIM Models Using a 3D Solids Modeling Wor...Integration of Utility Data into 3D BIM Models Using a 3D Solids Modeling Wor...
Integration of Utility Data into 3D BIM Models Using a 3D Solids Modeling Wor...
Safe Software
 
Oracle Cloud Infrastructure AI Foundations
Oracle Cloud Infrastructure AI FoundationsOracle Cloud Infrastructure AI Foundations
Oracle Cloud Infrastructure AI Foundations
VICTOR MAESTRE RAMIREZ
 
Creating an Accessible Future-How AI-powered Accessibility Testing is Shaping...
Creating an Accessible Future-How AI-powered Accessibility Testing is Shaping...Creating an Accessible Future-How AI-powered Accessibility Testing is Shaping...
Creating an Accessible Future-How AI-powered Accessibility Testing is Shaping...
Impelsys Inc.
 
Data Virtualization: Bringing the Power of FME to Any Application
Data Virtualization: Bringing the Power of FME to Any ApplicationData Virtualization: Bringing the Power of FME to Any Application
Data Virtualization: Bringing the Power of FME to Any Application
Safe Software
 
Soulmaite review - Find Real AI soulmate review
Soulmaite review - Find Real AI soulmate reviewSoulmaite review - Find Real AI soulmate review
Soulmaite review - Find Real AI soulmate review
Soulmaite
 
6th Power Grid Model Meetup - 21 May 2025
6th Power Grid Model Meetup - 21 May 20256th Power Grid Model Meetup - 21 May 2025
6th Power Grid Model Meetup - 21 May 2025
DanBrown980551
 
Boosting MySQL with Vector Search -THE VECTOR SEARCH CONFERENCE 2025 .pdf
Boosting MySQL with Vector Search -THE VECTOR SEARCH CONFERENCE 2025 .pdfBoosting MySQL with Vector Search -THE VECTOR SEARCH CONFERENCE 2025 .pdf
Boosting MySQL with Vector Search -THE VECTOR SEARCH CONFERENCE 2025 .pdf
Alkin Tezuysal
 
Domino IQ – What to Expect, First Steps and Use Cases
Domino IQ – What to Expect, First Steps and Use CasesDomino IQ – What to Expect, First Steps and Use Cases
Domino IQ – What to Expect, First Steps and Use Cases
panagenda
 
LSNIF: Locally-Subdivided Neural Intersection Function
LSNIF: Locally-Subdivided Neural Intersection FunctionLSNIF: Locally-Subdivided Neural Intersection Function
LSNIF: Locally-Subdivided Neural Intersection Function
Takahiro Harada
 
National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...
National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...
National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...
Safe Software
 
Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...
Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...
Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...
Anish Kumar
 
Mastering AI Workflows with FME - Peak of Data & AI 2025
Mastering AI Workflows with FME - Peak of Data & AI 2025Mastering AI Workflows with FME - Peak of Data & AI 2025
Mastering AI Workflows with FME - Peak of Data & AI 2025
Safe Software
 
AI Agents in Logistics and Supply Chain Applications Benefits and Implementation
AI Agents in Logistics and Supply Chain Applications Benefits and ImplementationAI Agents in Logistics and Supply Chain Applications Benefits and Implementation
AI Agents in Logistics and Supply Chain Applications Benefits and Implementation
Christine Shepherd
 
TimeSeries Machine Learning - PyData London 2025
TimeSeries Machine Learning - PyData London 2025TimeSeries Machine Learning - PyData London 2025
TimeSeries Machine Learning - PyData London 2025
Suyash Joshi
 
Developing Schemas with FME and Excel - Peak of Data & AI 2025
Developing Schemas with FME and Excel - Peak of Data & AI 2025Developing Schemas with FME and Excel - Peak of Data & AI 2025
Developing Schemas with FME and Excel - Peak of Data & AI 2025
Safe Software
 
Improving Developer Productivity With DORA, SPACE, and DevEx
Improving Developer Productivity With DORA, SPACE, and DevExImproving Developer Productivity With DORA, SPACE, and DevEx
Improving Developer Productivity With DORA, SPACE, and DevEx
Justin Reock
 
Jira Administration Training – Day 1 : Introduction
Jira Administration Training – Day 1 : IntroductionJira Administration Training – Day 1 : Introduction
Jira Administration Training – Day 1 : Introduction
Ravi Teja
 
What is Oracle EPM A Guide to Oracle EPM Cloud Everything You Need to Know
What is Oracle EPM A Guide to Oracle EPM Cloud Everything You Need to KnowWhat is Oracle EPM A Guide to Oracle EPM Cloud Everything You Need to Know
What is Oracle EPM A Guide to Oracle EPM Cloud Everything You Need to Know
SMACT Works
 
GIS and FME: The Foundation to Improve the Locate Process of Utilities
GIS and FME: The Foundation to Improve the Locate Process of UtilitiesGIS and FME: The Foundation to Improve the Locate Process of Utilities
GIS and FME: The Foundation to Improve the Locate Process of Utilities
Safe Software
 
Agentic AI: Beyond the Buzz- LangGraph Studio V2
Agentic AI: Beyond the Buzz- LangGraph Studio V2Agentic AI: Beyond the Buzz- LangGraph Studio V2
Agentic AI: Beyond the Buzz- LangGraph Studio V2
Shashikant Jagtap
 
Integration of Utility Data into 3D BIM Models Using a 3D Solids Modeling Wor...
Integration of Utility Data into 3D BIM Models Using a 3D Solids Modeling Wor...Integration of Utility Data into 3D BIM Models Using a 3D Solids Modeling Wor...
Integration of Utility Data into 3D BIM Models Using a 3D Solids Modeling Wor...
Safe Software
 
Oracle Cloud Infrastructure AI Foundations
Oracle Cloud Infrastructure AI FoundationsOracle Cloud Infrastructure AI Foundations
Oracle Cloud Infrastructure AI Foundations
VICTOR MAESTRE RAMIREZ
 
Creating an Accessible Future-How AI-powered Accessibility Testing is Shaping...
Creating an Accessible Future-How AI-powered Accessibility Testing is Shaping...Creating an Accessible Future-How AI-powered Accessibility Testing is Shaping...
Creating an Accessible Future-How AI-powered Accessibility Testing is Shaping...
Impelsys Inc.
 
Data Virtualization: Bringing the Power of FME to Any Application
Data Virtualization: Bringing the Power of FME to Any ApplicationData Virtualization: Bringing the Power of FME to Any Application
Data Virtualization: Bringing the Power of FME to Any Application
Safe Software
 
Soulmaite review - Find Real AI soulmate review
Soulmaite review - Find Real AI soulmate reviewSoulmaite review - Find Real AI soulmate review
Soulmaite review - Find Real AI soulmate review
Soulmaite
 
6th Power Grid Model Meetup - 21 May 2025
6th Power Grid Model Meetup - 21 May 20256th Power Grid Model Meetup - 21 May 2025
6th Power Grid Model Meetup - 21 May 2025
DanBrown980551
 
Boosting MySQL with Vector Search -THE VECTOR SEARCH CONFERENCE 2025 .pdf
Boosting MySQL with Vector Search -THE VECTOR SEARCH CONFERENCE 2025 .pdfBoosting MySQL with Vector Search -THE VECTOR SEARCH CONFERENCE 2025 .pdf
Boosting MySQL with Vector Search -THE VECTOR SEARCH CONFERENCE 2025 .pdf
Alkin Tezuysal
 
Domino IQ – What to Expect, First Steps and Use Cases
Domino IQ – What to Expect, First Steps and Use CasesDomino IQ – What to Expect, First Steps and Use Cases
Domino IQ – What to Expect, First Steps and Use Cases
panagenda
 
LSNIF: Locally-Subdivided Neural Intersection Function
LSNIF: Locally-Subdivided Neural Intersection FunctionLSNIF: Locally-Subdivided Neural Intersection Function
LSNIF: Locally-Subdivided Neural Intersection Function
Takahiro Harada
 
National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...
National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...
National Fuels Treatments Initiative: Building a Seamless Map of Hazardous Fu...
Safe Software
 
Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...
Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...
Scaling GenAI Inference From Prototype to Production: Real-World Lessons in S...
Anish Kumar
 
Mastering AI Workflows with FME - Peak of Data & AI 2025
Mastering AI Workflows with FME - Peak of Data & AI 2025Mastering AI Workflows with FME - Peak of Data & AI 2025
Mastering AI Workflows with FME - Peak of Data & AI 2025
Safe Software
 
AI Agents in Logistics and Supply Chain Applications Benefits and Implementation
AI Agents in Logistics and Supply Chain Applications Benefits and ImplementationAI Agents in Logistics and Supply Chain Applications Benefits and Implementation
AI Agents in Logistics and Supply Chain Applications Benefits and Implementation
Christine Shepherd
 
Ad

Aggregation Framework in MongoDB Overview Part-1

  • 2. What is Aggregation English Definition:- The act of gathering something together. Database Definition:- Aggregation are operations that process on the data sets, group some data records, do some computation on that records and return computed results.
  • 3. Aggregration Approach in MongoDB ● Aggregation Pipeline ● Map-Reduce ● Single Purpose Aggregation Operations ● Hadoop Connector
  • 4. Aggregation-Pipeline Similar to pipeline in UNIX. In Unix - Cat file.txt | grep abc | wc -l In MongoDB - Group Limit SortCollection Output
  • 5. MapReduce ● Two Phase process – Mapper & Reducer. ● Use JavaScript function for map-reduce job. ● Less efficient as it run over only one thread. ● Finalise stage to make final modification.
  • 7. Aggregarion Vs. MapReduce Easy to use, just need to use the build in operators. Complex and steep learning curve. Supports non-sharded and sharded input collections. Supports non-sharded and sharded input collections. Returns results inline. Return result in inline, new collection, merge, replace, reduce. Limited to the operators and expressions supported by the aggregation pipeline. Custom map, reduce and finalize JavaScript functions offer flexibility to aggregation logic.
  • 8. Single Purpose Aggregation Operation ● Applicable for single purpose of aggregation, like count, distinct, group. ● Limited scope as compared to aggregation pipeline and map-reduce. ● group does not support data in sharded collections, and result of operation should not be more than 16 MB.
  • 9. Aggregation Pipeline Operators ● $group ● $match ● $project ● $limit ● $sort ● $unwind ● $skip
  • 10. SQL to Aggregation Mapping Chart SQL Operators Aggregation Operators WHERE $match GROUP BY $group SELECT $project LIMIT $limit ORDER BY $sort SUM() $sum COUNT() $sum JOIN $unwind (Not exact operator as JOIN works, but unwinds the array embedded in the document)
  • 11. Examples SQL Query Aggregation Query SELECT COUNT(*) AS COUNT FROM EMPLOYEE db.employee.aggregate([ {$group: {_id:null, count: { $sum:1} } } ]) Explaination : Group by : on nothing Sum : Just add 1 to count field for each record SELECT SUM(SALARY) AS TOTAL FROM EMPLOYEE db.employee.aggregate([ {$group: {_id:null, total: { $sum:”$salary”} } } ]) Explaination : Group by : on nothing Sum: do the sum of value of salary field of each doc and put result in total.
  • 12. Example Cont. SELECT DEPARTMENT_ID, SUM(SALARY) AS TOTAL FROM EMPLOYEE GROUP BY DEPARTMENT_ID ORDER BY TOTAL db.employee.aggregate( [ { $group: { _id: “$department_id”, total: { $sum:”$salary” } }, { $sort: { total : 1 } } } ] ) Explanation: Group by : department, Sum : salary, Order by : total of salary for each department
  • 13. Example Cont. SELECT DEPT_ID, SUM(SALARY) AS TOTAL FROM EMPLOYEE WHERE AGE>25 GROUB BY DEPT_ID, HAVING TOTAL > 5000 db.employee.aggregate( [ { $match : { age : {$gt: 25 } } } { $group: { _id: “$dept_id”, total: { $sum: ”$salary” } } }, { $match: { total: {$gt:5000 } } } ] ) Explanation: Group by : department id, Sum : salary, Having: on total of each department salary
  • 14. Example Cont. SELECT DEPT_ID, SUM(SALARY) AS TOTAL FROM EMPLOYEE WHERE AGE>25 GROUB BY DEPT_ID, HAVING TOTAL > 5000 db.employee.aggregate( [ { $match : { age : {$gt: 25 } } } { $group: { _id: “$dept_id”, total: { $sum: ”$salary” } } }, { $match: { total: {$gt:5000 } } } ] ) Explanation: Group by : department id, Sum : salary, Having: on total of each department salary
  • 15. $unwind Operator Decompose the embedded array into flat document and relate each entry in the array with outer fields. Example :- { _id: “blog”, tags: [ “social”, “economic” ] } { _id: “blog”, { _id: “blog” tags: “social” } tags: “economic” }
  • 16. Mongo Aggregation Optimization MongoDB re-arranged the pipeline operations to optimize the aggregation performance. ● Pipeline Sequence Optimization. ● Projection Optimization.
  • 17. Mongo Aggregation Optimization MongoDB re-arranged the pipeline operations to optimize the aggregation performance. ● Pipeline Sequence Optimization. ● Projection Optimization.
  • 18. Pipeline Sequence Optimization ● $sort + $skip + $limit { $sort: { salary : -1 } }, { $sort: { salary: -1 } }, { $skip: 10 }, { $limit: 20 }, { $limit: 10 } { $skip: 10}
  • 19. Projection Optimization ● $project ● Reduce the amount of data passing through channels of operation and will help in performace improvement. ● Below example will only emit salary. db.employee.aggregate( [ {$match: {“name”: “xyz” } } { $project: { salary:1, _id:0} } ] )
  • 20. Restriction ● Output BSON document cannot exceed the 16 MB of data. If exceed will throw error. ● If single aggregation operation consumes more than 10 percent of system RAM, the operation will produce error.
  • 21. Aggregation Examples ● Download data from the below link: http://media.mongdb.org/zips.json ● Practice sample problem from the below links: http://docs.mongodb.org/manual/tutorial/aggregatio-zip-code-data-set/
  • 22. References ● http://docs.mongodb.org/ Good Examples:- ● http://rubayeet.wordpress.com/2013/12/29/web-analytics -using-mongodb-aggregation-framework/ ● http://derickrethans.nl/aggregation-framework.html ● http://architects.dzone.com/articles/using-mongodb -aggregation