Improving Security in CI/CD Pipelines (Review)

There is no CI/CD without git commit and git push. So, let us talk about what a CI/CD pipeline genuinely is. By automating the stages of app deployment, DevOps teams use CI/CD pipeline to deliver apps consistently. This procedure, which is started by a git push, aims to enhance software delivery throughout the development life cycle.

Before the introduction of automated testing, several developers working together on a project increased the likelihood that bugs would be pushed to production. Developers would then have to revert commits to find the bug and fix it. This isn’t scalable when a project consists of 10+ developers, but with CI/CD pipeline, projects with hundreds or even thousands of developers are feasible and fewer bugs make it to production.

The CI/CD pipeline, on the other hand, is a target of compromise and abuse. Because of its access to proprietary code, databases, credentials, secrets, and development and production environments, it is a tempting target for hackers. This is a growing concern for many DevOps-driven organizations.

This article will take you through a few preventative steps to help you keep your CI/CD pipeline secure.

Reduce risks with feature flags

New features are prime targets for security flaws, especially if they haven’t been thoroughly tested. Feature flags are frequently used by developers to add new features to applications while reducing the chance that those features will cause performance or stability problems. Developers can easily integrate the features into the codebase while still having the option to quickly disable them if necessary by using “flags” to turn the features on and off.

Issuing security flags can help developers reduce the risk of security issues because they can toggle features. Feature flags assist developers in continuing to improve their applications while keeping the associated risks in check.

You can create feature flags in SCA tools like Gitlab, connect them to a specific environment, and grant access to specific users. Tags in Semaphore can serve as feature flags in workflows, giving you more control. For every workflow, you can even whitelist the tags.

Note that feature flags are advisable in closed source software but not with open source. In open source feature flags could easily be compromised and this could result in a security issue. You can solve this by restricting access.

Restrict access

To control access to the CI/CD pipeline, create access control lists and rules; it should be immediately and easily obvious who has access to what, when, and how.

Every pipeline component and resource, regardless of whether it is role-based, time-based, or task-based, should be logged, monitored, and managed. Conduct routine audits to verify that unused machine or service accounts, as well as add accounts of former employees, are closed or have their access privileges revoked. Require multi-factor authentication for each user. For containers to be secure against nonhuman (bot) access, machine identity and authentication are essential.

Semaphore helps with this by providing an access security layer for forked pull requests in workflows. This greatly helps in securing open source contributions.

Issues with secret protection in CI/CD pipelines

Access to applications and services requires authentication credentials, such as usernames, passwords, API tokens, SSH keys, and encryption keys. Even developers who are dead serious about securely managing passwords, SSH keys, and other secrets in production environments, are tempted to cut corners with secrets management during the development and testing stages of the pipeline. However, as CI/CD pipelines consume more secrets, complexities arise, making it difficult to securely store, transmit, and audit secrets.

Semaphore has provided a way to deal with this problem.

For example, you could create a secret called Stripe Access Keys if you already have a set of environment variables (STRIPE PUBLIC KEY, STRIPE SECRET KEY) for accessing your resources on Stripe. After that, you can divulge that information to the teams that require access to the same resources.

The purpose of a secret is to reduce environment variable and configuration file duplication among your teams and projects. A person with an “admin” or “owner” permission level within the organization should set up several organization-wide secrets required for developing your projects. This is the ideal workflow for secrets. Members of teams with “admin” and “owner” permissions can share secrets with other teams after they have been set up. You must include it in your teams and attach it to your project to use entries from the secret.

Learn more about setting up secrets in Semaphore.

Debugging the CI/CD Pipeline

Although the aforementioned best practices help to avoid pushing bugs to production, the CI/CD pipeline itself is vulnerable to bugs due to the actions defined in your pipeline yaml file.

A pipeline can fail due to incorrect CI/CD logic. It becomes tedious when you have to repeatedly make changes to your CI/CD logic, push it up, wait for it to run, and it keeps failing.

How do you find out what happened when you run your pipeline and detect a flaw in your logic?

Your pipeline logic can be put into scripts, which you can then call from the pipeline yaml file.

In fact, Semaphore gives you the option to start on-demand virtual machines to explore the CI/CD environment, inspect log files, restart your jobs in debug mode, and access all currently running jobs via SSH.

Conclusion

The goal of developers should be to leverage practices that minimize the risk of active vulnerabilities reaching production environments in the first place. These actions consist of security tests, feature flags, etc. The more secure your CI/CD pipeline is, the fewer issues you’ll have to contend with once your code is in production. This will save you lots of time and headaches in the long run.