We cut our CI pipeline execution time in half (opens in new tab)

I think it’s that you are an old fuddy duddy :P

Actually, if you work with SMBs/enterprises, I agree with you on customer facing changes. In my past life we would ship very frequently (often more than once a day) but always had to feature flag changes that large clients might see or be affected by. Even something as simple as tweaking the layout of a core flow could cause support headaches and angry customers — customers worth 10s of thousands of dollars per month. Is it worth losing a customer to CD a new button placement?

This is considered the norm for high performing product teams in the modern day.

We keep customers happy because we push changes live incrementally, reduce our chances of major outages and improve our response time when they do occur.

This just sounds like Continuous Delivery. We never achieved it in my last job, so I can't speak from experience, but my understanding is that typically "deploy" is separated from "release" using feature flags of some kind.

Yep, it's part of our culture, we do many releases per day to constantly iterate things. Also as in other projects there are maintenance and bug fixing we want to bring to production as soon as possible.

Our context is the one of a startup that is constantly validating things, also in our context a release does not necessarily mean releasing to the users, sometimes stuff is behind feature flags or for beta testing.

Hey there! I'm an old fuddy duddy, too.

Continuous deployment has been around long enough that even IBM (remember never getting fired for buying IBM?) talks about it.

https://www.ibm.com/topics/continuous-deployment

"Dark deploys" and "feature flags" are often used to keep customers safe from incomplete features while still giving all of the advantages of CD plus allowing testing in production.

I'd never heard of Flagship, but this is a nice writeup on that (kudos, Flagship.io):

https://www.flagship.io/glossary/dark-launch/

Mh, I'm interacting with teams with wildly different release strategies and stability requirements at work.

One of the more fundamental things actually pushing towards faster releases is what I call the relativistic deployment speed. We have products that will need at least 2 months to get a remotely deployable version ready. The average fast hotfix usually takes more like 4 months until an installation on a prod system actually can start. Our fastest products can go from code to prod in like 15 minutes with the automated tests being the bottleneck.

This in turn shapes choices for the product managers, but also for security. If something like Log4shell hit these slow products, I'd have to plan to be vulnerable for two months at least, and usually more like 4 - 8 months depending on the customers. I have no choice, because that's their light speed of deployment. No code goes to prod faster than two months latency. That, quite frankly, fucking sucks.

Other products were much better in that situation. We were lucky to have the right devs around, but we went from the decision to emergency log4shell at an utmost risky speed to the first log4shell patches in prod of many within 30 minutes.

However, that's not the normal speed, and that's when you get into the second decision area. Given a lightspeed of deployment, how fast do you want to go?

Some of our possibly faster moving products are B2B products, with a lot of internal training for support and consulting going into a release, and also training at customers happening for larger customers. This means, product chooses to only release bigger changes and heavily customer-visible changes every 6 weeks. They could do this a lot faster, but they choose to slow down because it fits their customers well. And for example, december is usually frozen entirely because customers want to.

But then there is the third decision area. What happens if there is an entirely customer invisible change, such as an optimization in database handling, some internal metric generation for an optimization, or an internal change to prepare a new feature for the next scheduled rollout? And we have the tested, vetted and working option to just push that into prod without downtime, with also gives us opportunity to build experience with, and confidence into our no-downtime deployment system? I don't see a reason why I wouldn't exercise this daily at least once.

Read the State of DevOps reports over the years and you'll see why this is the direction we're all heading now. It turns out all that safety we thought we were building by making complex commit flows to multiple branches and environments was not only more complex than it needed to be but has also slowed us down and not made things better. Truly based development is back again and this time with data. Push early, push often, push small changes and iterate quickly. It's not just easier but it also seems to increase quality. (There are a lot of reasons why this turns out to be true. Read Accelerate. I won't do a better job explaining in a comment.)

That’s continuous delivery, right? You make great tests and you should feel comfortable releasing after review.

The short answer is "do as little as possible". What this means in practice is breaking down every step of CI, figuring out the dependencies for that step, and then ordering the graph of dependencies such that you start as much as possible as early as possible. This process also usually shows you where things are slow and what the critical path is.

Unfortunately, doing this in most CI services is actually quite difficult. It usually means a complex graph of execution, complex cache usage, and being careful to not re-generate artifacts you don't need to.

In my experience, building this, at a level of reliability necessary for a team of more than a few devs, is hard. Jenkins can do it reliably, but doing it fast is hard because the caching primitives are poor. Circle and Gitlab can do it quickly, but the execution dependency primitives aren't great and the caches can be unreliable. Circle also has _terrible_ network speeds, so doing too much caching slows down builds. GitHub Actions is pretty good for all of this, but it's still a ton of work.

The best answer is to use a build system or CI system that is modelled in a better way. Things like Bazel essentially manage this graph for you in a very smart way, but they only really work when you have a CI system designed to run Bazel jobs, and there aren't many of these that I've seen. It's a huge paradigm shift, and requires quite a lot of dev work to make happen.

Set up the environment on the runner, not in the job: https://docs.gitlab.com/ee/ci/runners/configure_runners.html...

At the very least, see if you can keep heavy dependencies on the local network rather than depending on the internet.

I'm not sure how to speed up the spin-up-a-container time (at least not without more details), but I have two suggestions that may help mitigate it. Based on your wording ("waiting for docker to spin up a container"), the second one may not be relevant.

## 1. Do more in the job's script

If you have multiple jobs that use (or could use) the same image, perhaps those jobs can be combined. It's definitely a tradeoff, and it depends on what you want from your pipeline. For example, normally you may have separate `build` and `test` jobs, but if they take, say (30s init + 5s work) + (30s init + 10s work), then combining them into a single job taking (30s init + 15 s work) _might_ be an acceptable trade-off. (These numbers are small enough that it probably isn't, but you get the idea.)

## 2. Pre-build the job's image

If your job's script uses an off-the-shelf image, and has a lot of setup, consider building an image that already has that done, and using that as your job's image instead. For example, you might be using a `node` image, but your build requires pulling translations from a file in an S3 bucket, and so you need to install the AWS CLI to grab the translation file. Rather than including the installation of the AWS CLI in the script, build it into the image ahead of time.

Make sure your docker build is being cached properly, and break infrequently running stuff into their own steps, then move them to the top of the docker file.

Crucially: Make sure that the large layers say they are "cached" when you rebuild the container. Docker goes out of its way to make this difficult in CI environments. The fact that it works on your laptop doesn't mean that it will be able to cache the big layers in CI.

Once you've done that, make sure that the CI machines are actually pulling the big layers from their local docker cache.

30-90 seconds to pull docker images for each run of a golang project's CI environment is too high. You might look into using "go mod vendor" to download the dependencies early in the docker build, then using a symlink and "--mod=vendor" to tell the tests to use an out-of-tree vendor directory. (I haven't tested this; presumably go will follow symlinks...)

My usual strategy is to ensure that the lengthy parts are executed only once. So for example one of the lengthy parts is environment setup for me too. So what I did is to put as much as possible on the docker image I build and then I start tests from image mostly ready to run. Of course something similar can be done during runtime. If starting the software you test takes long time, you could set it up only once and run multiple tests without tearing down the setup. Of course this has disadvantage of having possibly tainted environment and there is risk of making the tests depend on previous state. On the other hand this could also help discover problems that are hidden by always running tests on clean slate, so it's a tradeoff. And I have to note that I mostly do integration testing, so the long parts are probably in different places than for unit testing.

I had similar problems with CircleCI and it’s docker executor. We recently switched to GitHub actions and the following led to huge improvements:

- much faster network speeds. - We no longer run on the docker executor. Instead we run on Ubuntu. These boot in a second or 2 pretty consistently. - the bulk of our test suite was able to be pulled out of docker entirely (a lot of jest, and PHPUnit tests). - we have a bigger suite of E2E PHPUnit tests that we spin up a whole docker compose stack for. These are slower but still manageable.

Parallelism is key in all of this too. Our backend test suite has a full execution time of something like 250 minutes, but we just split it over a bunch of small workers and the whole thing completes in about 8 minutes.

Pulling snapshots helps, particularly with slowdowns over time. Pulling deps is a problem that deserves its own initiatives.

For me the controlling factor with build time and to a lesser extent production performance is to divorce visibility from vigilance. You can’t watch things 24/7 waiting to pounce on any little size or time regressions. You need to be able to audit periodically and narrow the problem to a commit or at least an hour in a day when the problem happened. Otherwise nobody will be bothered to look and it’s just a tragedy of the commons.

Graphs work well. Build time, test count, slow test count, artifact sizes, and so on.

How long are we talking? Are the containers getting pulled from somewhere across the internet and it’s a network bottleneck?

I noticed that many times using cache makes gitlab ci take longer than just fetching node dependencies again via npm install.

If you're mostly just compiling Go then why not cut out docker entirely? Just run your CI on bare metal.

How do you setup or provision your environment? And what does this environment look like?

It's not "pretty standard", but we're working towards it and it looks like a pretty great solution. Our problem is that CI job runners sleep most of the day (low number of commits), but then you have spikes where the jobs are waiting on each other and times get really long. Autoscaling sounds great - you can have lots of runners when you need them and only a single one (or maybe even none? not sure yet) otherwise.

Only if your company is "Cloud Native" and thus real concerned about paying for over provisioned compute.

Gitlab makes it pretty easy to just toss a ci runner process on a vm or a physical box. You can get real far with a couple rack servers and some xeons for < $1000. You do have to over provision if your work load is not very consistent ( and of course pay for the power and rack space, and someone to mind them from time to time).

If you have it, it’s awesome. You can get parallel execution of so much, spin up environments for each branch for QA and dynamic scans.

IMO it’s the optimal use case for K8s

Using kube for that is pretty fancy if you aren't already using kube elsewhere, but you don't just have a single Jenkins worker, you have multiple. All that kube is doing is giving a very convenient lever for autoscaling, but other platforms give you this lever as well. If you're not scaling Jenkins workers (or whatever) to match demand, even manually (spin workers down on weekends), you're wasting developer time, compute resources, or both.

Someone's got a new project for Q2 if they aren't doing this already - it's a pretty easy sell if you calculate out the time savings for developers during busy time of day + savings on spinning down compute resources in the middle of the night/weekends, and being able to put "I saved the company $X in idle compute and saved developers Y hours per day" on your yearly performance review looks pretty good.

Yeah... I don't know. We don't, but we have talked about it though, because the Azure pipelines are. just. so. slow. On the other hand, more complexity and Rube Goldberg-machinery is not something we long for.

I have started tinkering with Fastbuild, and preliminary testing makes it seem like to good to be true, or the best thing since sliced bread. I'm sure there are drawbacks somewhere, but it's really fast.

Then again, a big chunk of our pipelines is not actually the compilation, but stuff like downloading nuget packages, uploading artifacts and stuff, all of which are. very. very. slow.

Thanks for this comment. I guess there's sometimes we (developers) take things for granted when they are not, and that puts a lot of pressure on us instead of celebrating our wins.

I would change now "pretty standard" by "we don't invented the wheel" xD :pray:, in the end I wanted to mean we use existing tools and "just" put them together

Yes. In fact, it's standard enough that it's a little odd that they specify "autoscaling" and "21 nodes", when they could have simply said "we use the kubernetes executor".

Even if you are using SaaS GitLab, there are still good reasons to have custom runners, and kube is one option for running them.

Yep, default runners in most CI platforms shared resources so they are prone to produce flakiness (depending on your set up).

That was one of the reasons we ended up setting up our own runners. Didn't mention in the post but we use spot VM instances.

Interesting, how do you define CI in that case? IIRC, CI was originally defined as integrating continuously (i.e. daily or more frequently) and CD is delivering said code continuously. How does CI hurt productivity and how do you do CD without CI?

I'm really interested in different points of view.

I guess you mean the kind of trunk based development? But still some sort of CI happens, maybe locally.

Never worked in a different way than using a local / remote CI pipeline, that's why I'm curious.

CI is a prerequisite for CD.