Exploring MongoDB Aggregation Pipelines for Data Processing

What is 'Aggregation'?

What is a 'Pipeline'?

What role does they play in MongoDB databases?

How are they written?

Let's talk about it in this article from my own perspectives and understanding.👨‍💻

Starting as a developer, if you visit the aggregation pipeline documentation on MongoDB's official site here, it states that an aggregation pipeline consists of one or more 'stages' that process 'documents'.

In my opinion, this is the clearest and most concise explanation. To clarify, a document here is essentially the output of a query in JSON format, as MongoDB uses a JSON-based syntax.

For example:

db.orders.insertMany( [
   { _id: 0, name: "Pepperoni", size: "small", price: 19,
     quantity: 10, date: ISODate( "2021-03-13T08:14:30Z" ) },
   { _id: 1, name: "Pepperoni", size: "medium", price: 20,
     quantity: 20, date : ISODate( "2021-03-13T09:13:24Z" ) },
   { _id: 2, name: "Pepperoni", size: "large", price: 21,
     quantity: 30, date : ISODate( "2021-03-17T09:22:12Z" ) },
   { _id: 3, name: "Cheese", size: "small", price: 12,
     quantity: 15, date : ISODate( "2021-03-13T11:21:39.736Z" ) },
   { _id: 4, name: "Cheese", size: "medium", price: 13,
     quantity:50, date : ISODate( "2022-01-12T21:23:13.331Z" ) },
   { _id: 5, name: "Cheese", size: "large", price: 14,
     quantity: 10, date : ISODate( "2022-01-12T05:08:13Z" ) },
   { _id: 6, name: "Vegan", size: "small", price: 17,
     quantity: 10, date : ISODate( "2021-01-13T05:08:13Z" ) },
   { _id: 7, name: "Vegan", size: "medium", price: 18,
     quantity: 10, date : ISODate( "2021-01-13T05:10:13Z" ) }
] )

Source: MongoDB documentation

All the individual objects we are inserting into the 'orders' database are documents.

Now, let's discuss the next term in the definition: 'Stages.' Stages are the individual operations we perform while writing our aggregation pipeline. Our aggregation pipeline consists of an array of individual objects, and these objects are the stages that perform our desired operations on the documents. These operations can include filtering data by some value, grouping data based on a condition, joining two different database models based on a rule, adding fields, and more. You can read about them in the documentation here.

Here's a example of such stages:

{
    $match:{size:'medium'} //filter by size
},
{
    $group:{ _id: '$name',}//group by name
},
{
    $lookup:{//performs a left join
           from: 'authors', //collection to join
           localField: 'authors._id',//field(id) from input documents,
           foreignField: '_id' //field(id) from documents of 'authors'
           as:'test' //output array field (basically like an alias) 
           },
{
     $pipeline: // used to write further sub-pipelines 
                //(simply a pipeline inside a pipeline)
}

Above are a few examples of stage methods. The 'lookup' method might seem confusing at first, but if you're familiar with SQL, it will be easier to understand. The names 'localField' and 'foreignField' are similar to Primary and Foreign Keys in SQL, so you can relate to them easily.

Similarly, all these stage methods have a lot in common with certain SQL queries, which you can explore further in MongoDB's documentation.

Conclusion

These are some basics of understanding and writing an Aggregation Pipeline in MongoDB. If you have experience with SQL, the concepts are easier to grasp, though the syntax may differ. For others, it becomes easier with practice. These concepts are best learned by coding and through personal experience.😊

So, that's it. I shared what I have learned and understood about the topic so far.

All feedback is welcome in the comments. 😀