CI/CD Pipelines You Will Love

These are some of the opinions I have formed through years of working with CI/CD pipelines. I have worked primarily with GitHub Actions, so that is what I will focus on. My philosophy on pipelines revolves around these core principles:

• Developer experience is king. Optimise accordingly.
• Engineers follow the path of least resistance.
• Given the same input, a pipeline run that succeeds today must also succeed in a year.

Pin Dependencies

When specifying dependencies in an action, pin them to a specific version. This goes for not only downstream actions, but also the runner itself as well as any packages manually installed:

name: Lint
on:
  push:
jobs:
  api:
    name: API
    # Pin the runner itself.
    runs-on: ubuntu-24.04
    steps:
      - name: Checkout
        # Pin the downstream action.
        uses: actions/checkout@1e31de5234b9f8995739874a8ce0492dc87873e2  # v4
      - name: Install vacuum
        # Pin manually installed packages.
        run: |
          npm i -g @quobix/vacuum@v0.20.4
      - name: Run vacuum
        run: |
          make lint-api

Note that the downstream action is pinned to a commit SHA rather than a tag. This is to enhance security. By default, tags are mutable and can be modified to point to a compromised commit¹. GitHub’s immutable releases feature can also be used to ensure immutability, but this feature is not yet widespread.

Few things are more frustrating than a pipeline that passed yesterday suddenly breaking despite there being no changes to it. Pinning dependencies helps prevent this.

Pipelines Are Software

Debugging embedded Bash code inside the conditionally triggered step of a pipeline instantly takes me back to Ryan Dahl’s excellent essay I hate almost all software. Bash code brings me no joy, and I certainly don’t want to understand some provider’s bespoke pipeline syntax. The mere sight of the tiny scroll bar on the syntax page for GitHub Actions makes me shudder.

I can tolerate pipelines that set up the necessary environment and then call a make command before exiting, but anything more complex should be extracted to a separate script, ideally one written in a real programming language².

Extracting pipeline functionality often takes a bit longer than writing it inline, but recall that we are not optimising for implementation time³. Software breaks all the time, and it is significantly easier to debug a pipeline that can be run locally in a familiar programming language. Another benefit of this approach is that it makes it easier to migrate pipelines across providers, but this is a happy accident and not something I deliberately aim for.

Run Often

GitHub Actions support myriad options for conditional triggers. I tend to avoid these. Instead, I prefer to implement the conditional check in the pipeline’s code itself and exit as late as possible. This exercises as much of the pipeline as possible as frequently as possible.

Consider a repository that has an OpenAPI YAML file. A pipeline is set up to lint this file, but only when the file changes:

name: Lint
on:
  push:
    paths:
      - 'openapi.yml'
jobs:
  api:
    # Run the linting job.

This approach has several issues. First, changes to the openapi.yml file are not the only way the file can become invalid. What if the linter rules change? Second, if something did cause the action to start failing, then it would take until the next change to the openapi.yml file for the failure to be detected.

In practice, I have found it difficult to predict the circumstances under which a pipeline can fail, and that conditional triggers often open the door for failures that are caught only long after they have been introduced.

Running as much as possible of all pipelines on every push shifts the process left. Following this advice will increase pipeline costs, but the reduction in friction is worth it. At an hourly rate of $50, a single engineer debugging for an hour costs the same as 25,000 GitHub Action minutes!

Pipelines Are Not a Secret Store

Pipelines often need access to secrets, and it seems sensible to store these in the GitHub repository and then reference them in the pipeline. I have found this not to be the case. It is not possible to see the value of a repository secret after setting it, so it must be stored in a secondary location as well. This is problematic because the secret no longer has a single source of truth. When the secret is rotated, it must be rotated both in the secret store and in every GitHub repository that uses it. This can quickly become a nightmare as there is no guarantee that repositories use the same name for the secret, so figuring out which repositories need to be updated can be nearly impossible.

A scenario I have found myself in more than once is debugging a pipeline that has suddenly started failing. My suspicion typically falls on an expired secret⁴, but the low visibility of repository secrets makes it difficult to confirm this. The secret is likely stored in at least one other place, but there is no way for me to figure out where. This leaves me in an unfortunate situation where I can’t see whether the secret has expired, if it was already rotated, what the latest value is or if the repository has been updated to use the latest value.

Instead of storing secrets in their repository, pipelines should load them from the authoritative secret store at runtime. For example, hashicorp/vault-action can be used to load secrets from Vault. Doing this raises visibility and solves the issues mentioned above. Secrets are now stored only in one place, and inspecting the pipeline definition allows one to see where the secret is loaded from.

Naturally, the pipeline may still need to have access to a secret used to authenticate to the secret store. If possible, connection to the secret store should be established using a trust relationship (Vault supports this with GitHub OIDC Tokens), but even if this is not possible, storing a single secret in the repository is vastly preferable to storing all required secrets.