Securing Bazel's Module Registry

As a developer, I view most posts about Supply-Chain Security with skepticism. Fear, uncertainty, and doubt (FUD) are a great way for security vendors to get leads and make sales, whether or not the underlying problem represents a real vulnerability.

But a couple weeks ago, there was yet another wake-up call: a vulnerability in tj-actions infected an unknown number of open-source projects, whose CI pipelines probably exposed tokens that can be used to hijack their projects.

Like the xz vulnerability last year, this happened in an OSS repo where a tiny number of individuals (often hobbyists) are responsible for holding up the security posture of enterprise users. And many Bazel rulesets are similarly-staffed projects! Google is transferring repositories to the Linux Foundation (under the bazel-contrib GitHub org) and this comes with no resources for maintenance.

What would happen if a Bazel ruleset author leaks a Personal Access Token? Rules do things like download toolchains, and invoke them to produce your build outputs. A malicious release artifact could easily inject code into the binaries you build and ship. This attack vector is outside of your code, and outside of the third-party packages that your security scanner runs on. I don’t know of any companies who are taking steps to prevent this today.

What can you do about it?

Whenever taking a new dependency, an enterprise security team will scan the sources for exploits, because they cannot trust the maintainers have a security posture that meets their requirements. When the dependency is updated by dependabot or renovate, hopefully they’re reviewing the delta to the sources since the prior trusted version.

It’s convenient that rulesets (and C++ packages) are published on the Bazel Central Registry so users don’t have to vendor the sources of all the modules they depend on. However as a consequence of GitHub’s checksum stability outage two years ago, Bazel modules generally publish a release artifact, which is constructed from the sources. How can an enterprise consumer know whether that artifact is truly built from the sources they’ve reviewed?

This is the assurance we can get from a framework like https://slsa.dev/. Thanks to a grant from Google and a bunch of help from Appu on the Distroless team, Aspect has upgraded the supply-chain for Bazel rulesets to optionally include an attestation, which is a cryptographic proof-of-trust. It allows an end-user of the module to place their trust in the build system vendor (GitHub in this case) that the build artifacts are provably constructed on a secure machine, given the sources and build scripts that exist in the repository for that release.

Attestations of BCR modules

As an example, let’s look at https://registry.bazel.build/modules/aspect_rules_lint/1.3.4. To be secure, one should follow the “Release Notes” link to navigate to the GitHub repo, and use the “Full Changelog” link to scan through the source code changes since the prior release we trusted. Okay, nothing looks malicious. How can we trust the release artifact on BCR?

That release also has three new files, with a .intoto.jsonl extension. These are attestations of provenance. What does that mean? Since the release was built on GitHub Actions, using GitHub-hosted runners, GitHub is providing a proof that the release artifact is constructed from the sources in the repo, using the workflow definition in the repo.

GitHub CLI provides the attestation verify command. If we download the release artifact we can try it out:

% gh attestation verify ~/Downloads/rules_lint-v1.3.4.tar.gz --owner aspect-build --signer-repo bazel-contrib/.github       
Loaded digest sha256:a7dfbfe0aa2fb960911a1589fa2ebc4f9fd0e25b0090edb54a9dfac73fdd6444 for file:///Users/alexeagle/Downloads/rules_lint-v1.3.4.tar.gz
Loaded 1 attestation from GitHub API

The following policy criteria will be enforced:
- Predicate type must match:................ https://slsa.dev/provenance/v1
- Source Repository Owner URI must match:... https://github.com/aspect-build
- Subject Alternative Name must match regex: (?i)^https://github.com/bazel-contrib/.github/
- OIDC Issuer must match:................... https://token.actions.githubusercontent.com

✓ Verification succeeded!

The following 1 attestation matched the policy criteria

- Attestation #1
  - Build repo:..... aspect-build/rules_lint
  - Build workflow:. .github/workflows/release.yml@refs/tags/v1.3.4
  - Signer repo:.... bazel-contrib/.github
  - Signer workflow: .github/workflows/release_ruleset.yaml@refs/tags/v7.1.0

A similar verification is built-in to the BCR presubmit process as well, ensuring that the attestations in the repository (https://github.com/bazelbuild/bazel-central-registry/blob/main/modules/aspect_rules_lint/1.3.4/attestations.json) may be trusted. Ultimately we expect that the Bazel client itself will transparently verify modules as they are downloaded, which will also support private registries.

Rolling out to the ecosystem

We didn’t just make this work for rules_lint. The changes needed have been made to the Publish to BCR helper, which is now a reusable workflow rather than a GitHub App. That means we’ll soon be able to trust releases from all the rules that reuse this release and publish workflow.

Thanks to Derek for doing the engineering work! If you’re a module author, follow that link to the docs for Publish to BCR.