Andrew Southall | Pipelines: A Bundle of Opinions

Continuous Integration / Deployment / Development / Delivery are high level objectives found at most organisations. The majority of SoWs that I’ve worked to included CICD implementation or the implementation of continuous CICD improvement which is continuously doing whatever-is-being-done. My experience has been that firms just want better and more reliable pipelines. Everyone is sick of seeing pipelines mysteriously fail.

What is a pipeline ¶

Pipelines are exactly what they sound like, they’re delivery mechanisms that take a known input from a known input point and deliver it to a known exit point. Along the way certain elements of the input can be managed, but ultimately what you put in is what you should get out the other end. This makes it different to say plumbing, where water enters a plumbing system and can be heated, pumped, mixed with, filtered or transformed into who knows what. Sewage plants, breweries, residential homes and farms all have plumbing. Oil refineries have pipelines. Crude oil enters and crude oil exits with the value obtained from the moving a lot of crude oil. Your pipelines should be doing heavy lifting and should shy away from anything more complex.

An Example

Let’s take the pipeline for this website which is managed by drone. Slightly abbreviated pipeline code is below:

kind: pipeline
type: docker
name: dev

steps:
- name: Updating prod
  image: smasherofallthings/hugo
  pull: if-not-present
  environment:
  commands:
  - npm run build-prod
  - aws s3 sync public/ s3://southall.solutions --size-only --acl public-read
  - aws cloudfront create-invalidation --distribution-id <SNIP> --paths '/*'
  when:
    branch:
    - master

trigger:
  event:
    - push
    - cron

This pipeline has a single step triggered once every week and whenever I add code to the repository:

Build the front end code (JS & CSS) using the fronted libraries (npm run build-prod)
Build the content (HTML) using Hugo (npm run build-prod)
Uploads the content to AWS (aws s3 sync...)
Tells the CDN to invalidate all content (aws cloudfront...)

That’s it, there’s no need for the pipeline to do anything extra. The steps that the pipeline follows are the steps that I want followed for every commit to the repository. Every time I add an article or every time I update the styling, I want the above steps to run. The pipeline runs weekly, so if an article is configured with a publish date, and the pipeline runs after that date then the above steps will need to happen. In all situations, the front end needs building, the content needs building, the results need uploading to AWS and the CDN needs flushing out. If there wasn’t a pipeline then I’d still need to do all this, they’re required steps at all times.

If there’s an well drilling for oil the crude needs transporting, pipeline or not. If there’s code for a website the site needs deploying, pipeline or not.

This site has additional tools such as spellcheck and lint scripts. The pipeline doesn’t run these tools. It could run those tools, but only if I want a spellcheck and a lint run on every single commit. I’d prefer those to be run manually as I don’t want the pipeline failing because it didn’t ‘know’ the word Terraform.

The above is a true, certified-classic pipeline that leaves the intelligent work to better equipped entities. It doesn’t try to think ahead like checking that the CDN has flushed entirely or that I’ve paid the AWS bill. It doesn’t attempt to check website versions or update any libraries. All it does is take the code repository and the docker image, runs the commands and out the other end comes this website. In other words, it takes known inputs, a known destination and a delivers a set of known outputs. If the pipeline blows up or breaks, it just shuts down and better entities can handle the problem.

What is not a pipeline? ¶

A pipeline stops being a pipeline when:

It starts intermingling with other pipelines
It starts pulling or accepting inputs from outside the pipeline
It has multiple entry points
It has multiple exit points

These events result in plumbing, also known as orchestration. Orchestration is useful, but it shouldn’t be intermingled into a single pipeline. Orchestration sits over the top and should be treated as a more invested task.

For the above pipeline to run successfully there needs to be an AWS bucket and the CDN provisioned. If we didn’t have this bucket and CDN distribution then we wouldn’t have our known destination to hook the pipeline up to. So before the pipeline can work, I had to deploy out the bucket and CDN. Being realistic, that infrastructure is deployed once and then left alone, hence it’s not part of the continuous delivery of this site and therefore isn’t in the pipeline; we don’t need to continuously deliver a bucket and a CDN, we do it once and connect up the pipeline for the continuous bits.

I did the bucket and CDN deployment via Terraform so I can track everything and deploy it all easily without clicking around in a console. If I included that Terraform code in the site’s pipeline then I’ve added intermingled the setup of the pipeline with the usage of the pipeline. I’m needing the pipeline installer (Terraform) to come back and check that the pipeline is still installed every time I want to flush something through said pipeline (release an article). Sure, you can do that if you want to be thorough, but you bring all the baggage of Terraform and the underlying AWS API to the deployment of a web page.

If I had included Terraform in the above pipeline, then every single deployment to this website from April 2023 onward would have failed as AWS deployed out breaking changes to S3. In practical terms I wouldn’t be able to flush things through my pipeline as the installation is now out of date. It would be functionally fine but Terraform would have failed for unrelated reasons.

I recommend reducing pipelines to only perform known automated tasks that are able to fail but unable to deviate. Pipelines can explode, leak, get blocked, whatever, that’s all fine. What they shouldn’t ever do is take the same inputs and result in different outputs.

When does that actually happen though? ¶

All the time:

Updating packages inside of a pipeline - if you’re not pinning packages then you’re pulling in data from elsewhere. What will you get at the end? We can only hope
Database updates, upgrades or migrations inside of the pipeline - if you’re not controlling the data coming in then you’re hoping whatever data you do pull is fine. Don’t put code in at one point and let the pipeline pull whatever data from wherever database without defining and verifying that data
Using prebuilt pipeline “actions” - can you see what code is being executed? How do you know what it’s doing?
Implicit variables and controls - do you inherit variables or settings from outside the pipeline that are not explicitly set or provided to the pipeline? You’re flying blind

What’s actually bad about plumbing rather than pipelines? ¶

Everything is fine until it goes wrong. When things go wrong in a pipeline, you’re grabbing your tools and slicing up the pipe to investigate and remediate. This may mean rebuilding the pipeline steps in a testing environment and running them step by step. If your pipeline has unknown inputs then you’re going to spend time finding out what those inputs are.

Also, you’ve called it a pipeline, so everyone else is going to keep throwing things into it whilst you speedrun a fix. You’ve called it a pipeline, so they’re gonna treat it like one whether it’s working or not.

Simple automation is far easier to replicate in a different environment, different CI/CD tool or even locally should there be any issues. Replacing a pipe is far faster and far quicker than finding a problem that could be anywhere in the plumbing.

Additionally, if you need to swap tools you’re able to do so. GitHub actions are the most opinionated CI/CD tool I’ve come across. Most pipelines built for GitHub Actions are completely unmigratable. Should GitHub Actions be down you’re needing to sit tight or start a rewrite.

If you’re happy with your plumbing then rock on. If not, stick to simple pipelines.

How do I turn my plumbing into pipelines? ¶

Capture and gatekeep all inputs. If your pipeline needs credentials, then you’re explicitly checking for those credentials and reporting which credentials you have. If your pipeline needs certain variables, files or other settings to function then those are also explicitly checked for and reported. You want your pipeline logs to start with a full list of inputs & variables and followed by explicit commands. Now you know exactly what’s going on and can debug or replicate that automation elsewhere.

Don’t try to be smart. Machines are dumb. Pipelines are dumb. Smart devices are locking people out of their homes or setting the thermostat to 40°C. Dumb bridges are still standing and dumb metal tubes are still pumping crude oil over thousand of miles. Dumb is reliable.

The easy approach is to build your pipeline as local automation. Make, Ansible or other tooling is ideal. Shell scripting can handle simple steps elegantly. If you need to check for required environment variables then it’s as simple as:


reqVars=()
reqVars+=('DockerUsername')
reqVars+=('DockerPassword')
reqVars+=('ImageName')

for reqVar in "${reqVars[@]}"; do
  if [[ -z "${!reqVar}" ]]; then
    logAndExit "Did not find mandatory environment variable ${reqVar}! Exiting..."
  fi
done

It should go without saying but use a container with the code base locally mounted. Run your automation, test, alter, confirm, tear down and run end to end. Congrats, you’ve got a pipeline that you can run locally and from CI/CD tooling.

What about things that have to have multiple inputs? I still need to do data migrations ¶

Don’t run them in a pipeline. Do whatever you need to do but don’t wrap that up in a pipeline ran out of Jenkins, GitHub Actions, Drone, whatever. You’re just putting a barrier in front of you and automation. Data migrations are involved work and should get given the time and attention deserved rather than handed off to some pipeline.

Pipelines are dumb. Migrating and manipulating data is not always dumb. A dumb pipeline running a non-dumb set of data operations will disappoint. A disappointing data migration is something worth avoiding.

Slap the code into a mounted volume on a container and run/test/deploy it directly.

OK I get it ¶

Nice. I’m going to commit this article and push up. The pipeline for the site is going to do all the stuff I just wrote about. I’m not gonna watch. I’m going to let the pipeline handle the heavy lifting and spend my time on better things.

Pipelines: A Bundle of Opinions Bonus advice on outmanoeuvring disappointment