Monorepo vs. Polyrepo in GitLab CI/CD: Concepts and Best Practices

In modern software development, teams often must decide between two common repository management strategies: Monorepo and Polyrepo. This decision impacts how code is managed and how Continuous Integration and Continuous Deployment (CI/CD) pipelines are structured.

Let’s explore the key concepts of Monorepo and Polyrepo, how they affect GitLab CI/CD pipelines and the best practices for using the extends keyword in a Monorepo setting and GitLab templates in a Polyrepo environment.

Monorepo: A Unified Repository Approach

A Monorepo involves managing multiple projects or services within a single repository. This approach is common in large-scale software systems where shared code and tight integration between components are essential.

In GitLab CI/CD, Monorepo pipelines need to manage multiple projects or microservices, which can become complex if each service has its own distinct pipeline configuration. To simplify the pipeline setup and avoid duplication, GitLab provides the extends keyword, which allows you to reuse common job configurations across multiple jobs within the same .gitlab-ci.yml file.

Using the extends Keywords in Monorepo Pipelines

The extends keyword helps DRY (Don't Repeat Yourself) principles by letting you define common jobs or configurations once and reuse them in multiple places. This is especially useful in a Monorepo setup where services may share similar build, test, or deploy steps.

Here’s how to use the extends keyword in a Monorepo pipeline:

Example:

# .gitlab-ci.yml

# Define common stages and configurations
.default-job-template:
  script:
    - echo "This is a common step for all jobs"
  tags:
    - docker
  retry: 2

stages:
  - build
  - test
  - deploy

# Extend the common template for service 1
service1-build:
  extends: .default-job-template
  stage: build
  script:
    - echo "Building Service 1"
    - ./build-service1.sh

service1-test:
  extends: .default-job-template
  stage: test
  script:
    - echo "Testing Service 1"
    - ./test-service1.sh

# Extend the common template for service 2
service2-build:
  extends: .default-job-template
  stage: build
  script:
    - echo "Building Service 2"
    - ./build-service2.sh

service2-test:
  extends: .default-job-template
  stage: test
  script:
    - echo "Testing Service 2"
    - ./test-service2.sh

Key Points:

• Reusable Configurations: Define common jobs or steps once and extend them across multiple services. This reduces redundancy.

• Centralized Management: Managing pipeline configurations for multiple services in a Monorepo is more straightforward as everything is in one place.

• Better Consistency: Ensures service consistency using shared stages like build, test, or deploy.

Polyrepo: A Decentralized Repository Approach

In contrast to a Monorepo, a Polyrepo structure involves separate repositories for each service or project. This approach is functional when services or projects are more independent or when different teams work on isolated parts of the system.

However, with Polyrepo, you might face challenges maintaining consistent CI/CD pipeline configurations across multiple repositories. Manually duplicating the same .gitlab-ci.yml configurations across all repositories would be inefficient and error-prone. This is where GitLab Templates come in handy.

GitLab Templates in Polyrepo

GitLab allows you to create reusable pipeline configurations by defining templates. These templates can be stored in a central repository (or group-level templates) and shared across multiple repositories, making maintaining consistency in a Polyrepo setup easier.

Example: Creating a Reusable GitLab CI Template

1. Create a Template Repository:

First, create a repository that holds your shared templates.

Template repository structure:

template-repo/
  ├── .gitlab-ci.yml
  └── ci-templates/
      ├── build-template.yml
      ├── test-template.yml
      └── deploy-template.yml

2. Define the Templates:

In the template repository, create a file ci-templates/build-template.yml that contains a reusable build job:

# ci-templates/build-template.yml

.build-template:
  script:
    - echo "Running build step"
    - ./build.sh
  only:
    - master
  tags:
    - docker
  retry: 2

Similarly, create a test-template.yml and deploy-template.yml with appropriate steps.

3. Use the Templates in a Polyrepo Project:

In each Polyrepo, you can include the template files using GitLab’s include feature.

Example: Including Templates in a Polyrepo .gitlab-ci.yml file:

# .gitlab-ci.yml in service1 repository

include:
  - project: 'group/template-repo'
    file: '/ci-templates/build-template.yml'
  - project: 'group/template-repo'
    file: '/ci-templates/test-template.yml'

stages:
  - build
  - test
  - deploy

build-job:
  extends: .build-template
  script:
    - echo "Building Service 1"
    - ./build-service1.sh

test-job:
  extends: .test-template
  script:
    - echo "Testing Service 1"
    - ./test-service1.sh

Key Points:

• Consistency Across Repositories: Using GitLab templates ensures that all repositories have consistent pipeline configurations.

• Centralized Management: You can update the template once in the central repository, and all Polyrepos using it will automatically get the updates.

• Simplified Maintenance: Managing pipelines across multiple repositories becomes more manageable and less error-prone as configurations are defined in a single location.

Jenkins Shared Library vs. GitLab Extends & Templates

Jenkins' Shared Library has similarities with GitLab CI/CD’s extend keywords and Templates, as all three mechanisms are designed to reuse code, centralize configuration, and promote consistency across pipelines or jobs. However, each operates within its own system and has unique features tailored to the respective CI/CD platform.

Find below a quick explanation and comparison table:

• Jenkins Shared Library: A way to share common pipeline logic across multiple Jenkins jobs by writing Groovy scripts that can be reused. It allows Jenkins pipelines to import shared functions, variables, and stages from a central repository.

• GitLab extends: In a Monorepo, the extends keyword helps reuse job configurations across multiple jobs within a single .gitlab-ci.yml file. It reduces redundancy by allowing jobs to inherit configurations from a predefined job.

• GitLab Templates: In a Polyrepo setup, GitLab Templates are used to share common pipeline configurations across multiple repositories by importing them from a central location.

Comparison Table

Key Similarities:

• Reusability: All three mechanisms focus on code reuse, allowing developers to avoid duplicating pipeline logic across jobs or repositories.

• Centralized Management: Both Jenkins Shared Libraries and GitLab Templates allow centralized configuration, with any changes affecting all jobs that depend on them.

• Customization: Jenkins libraries, GitLab extends, and Templates all allow individual jobs to inherit common logic while customizing or overriding specific steps as needed.

Key Differences:

• Scope: Jenkins Shared Library works across multiple Jenkins jobs, while extends is limited to a Monorepo. GitLab Templates, on the other hand, can work across multiple repositories.

• Language: Jenkins relies on Groovy for shared libraries, whereas GitLab CI/CD is purely YAML-based.

• Implementation: Shared Libraries use code functions in Groovy, while GitLab extends and Templates rely on YAML-based inheritance or inclusion mechanisms.

Conclusion

Both Monorepo and Polyrepo strategies offer unique advantages, and GitLab CI/CD provides powerful features to manage pipelines effectively in both scenarios.

• In a Monorepo setup, the extends keyword is invaluable for reusing common pipeline configurations across multiple services within a single repository. This simplifies pipeline management and ensures consistency across services.

• In a Polyrepo setup, GitLab Templates offer a way to share and reuse pipeline configurations across multiple repositories, making it easier to maintain consistency and avoid duplication.