MongoDB Pipelines

Why Are MongoDB Pipelines Used?

MongoDB pipelines are used to perform complex data manipulations and aggregations directly within the database. They allow you to filter, sort, group, reshape, and transform documents in a collection without pulling large amounts of data into an application for processing. This leads to more efficient data handling and reduces the data transfer between the database and the application.

Schemas

We make Schemas to give our data a structure and the data we store into our database is in the format we initialise in the first place.

const userSchema = new mongoose.Schema({
    userName: { 
            type: String, 
            required: true 
             },
    email: { 
            type: String,  
            required: true 
           },
    phoneNo: { 
            type: Number, 
            required: true 
             }
}, { timestamps: true });

There are multiple Schemas for different things, and when we need to pass one schema's value to other schema, then we use mongoDB aggregation pipelines.

Aggregation Pipelines

It connects to different Schemas for the exchange of valuable data, it has many Stages like $lookup, $match, $addFields etc.

{
        $match:{
            username:username.toLowerCase()
        }
    }

$match stage: The $match stage filters the documents in a collection to allow only those that match specified criteria to pass through to the next stage of the pipeline.

{
  $addFields: {
    <newField>: <expression>,
    <existingField>: <expression>
  }
}

$addFields: sometimes a situation arises where we have to create new data feilds in our documents, we can directly create them using $addFeilds stage

Benefits of MongoDB Pipelines

1. Efficiency: By processing data directly on the server, MongoDB pipelines minimize the amount of data transferred over the network and reduce the load on client applications.

2. Powerful Data Aggregation: The Aggregation Framework offers a wide range of operations, including filtering, grouping, sorting, reshaping, and computing aggregations, allowing for comprehensive data analysis and reporting.

3. Flexibility: MongoDB's document model and the Aggregation Framework's flexibility make it easier to handle diverse data and complex queries compared to traditional SQL databases.

4. Scalability: Pipelines are designed to efficiently handle large datasets, taking advantage of MongoDB's horizontal scaling capabilities.

5. Real-time Data Processing: Pipelines can be used for real-time data transformations and analytics, which is beneficial for applications requiring up-to-the-minute insights.

Refer to this video :

Also refer to official documentation of mongoDB for better understanding:

MongoDB Pipelines