I meant to make a blog post about this, but here's a good a place as any: GitLab absolutely innovated many hard parts of CI/CD as a platform-native piece, but it feels like they lose slightly to GitHub on what GitHub does best -- social virality for developers. The problem with losing slightly there is that the advantage compounds; if developers find it easy to make and share, then they make and share which makes more people make and share and generates tons of value for a platform.
It may not be safe, but being able to:
- uses: some-developer/some-repo@some-branch
No one would accuse GitLab of being a "copy" in recent memory like they did in the bad old days, but IMHO they need to just copy the shit out of when GitHub gets social things (like this) right.
We know how to build, package, and release our software, and at work I firmly support that we write those operations as standalone scripts that in principle could even run in the developer's machine itself. Moving them to CI is just changing the machine that will run the scripts, and that's where I draw the line of how much coupling to accept with any given runner: run scripts, inject secrets, and maybe accumulate artifacts between jobs.
There are a myriad of things done with Actions, such as preparing NPM, or running Docker containers. I see no point in them, you should be able to manually run the same commands, one by one, locally on a VM or empty Docker container, so at that point why not write them as a script. Use CI as a dumb shell runner. Getting to depend on all those small Actions saves 5 minutes today, only to make migrations immensely painful tomorrow.
This may be a tradeoff you're ok with, but for a lot of people, it's not.
I like CI for secrets management, scrpted pipelines and deploying. And having one button do do all of that. But all the extra junk people do with it seems like another thing to have break.
That said it is nice (if the scripts are simple) to do things like machine set up very easily.
Oh god, if this is the best CI in the game, I don't want to be part of this game anymore. I work with (as in write pipeline code for) Gitlab CI almost every day and it's absolutely horrible. Not only does YAML lack any type safety whatsoever, but every day I run into yet another open ticket in the Gitlab issue tracker because using a declarative (YAML-based) approach (as opposed to an imperative one) essentially means that the Gitlab devs need to think of every possible use case beforehand, so as to be able to define a declaration & write an implementation for it. Clearly this is impossible, so now I'm banging my head against a wall on the daily.
If your CI process is so complex that YAML + arbitrary code doesn’t work, you might want to get that checked, it’s not normal
We did evaluate Gitlab CI but went with Drone. Gitlab CI is not a top 5 CI vendor IMO.
The latest redesign where they grouped all the sidebar links into even more categories just made it ludicrously worse.
As someone who only used GitLabs CI briefly when it initially launched, what hard parts did they innovate on exactly?
As far as I could tell, it's a run-of-the-mill CI/CD platform, for better or worse, but nothing I'd call "innovative".
But again, maybe since the first time I tried it when it launched, it has changed, and I missed something really cool they did.
My experience is limited to GitHub Actions, GitLab CI, and Jenkins. I've never been in an organization that managed to crack the code on running Jenkins in a sane way (always a poorly maintained, bloated mess of plugins, unfortunately) so I can't say I count Jenkins as a contender in this conversation. One could argue it's unfair to discount Jenkins because of organizations that manage it poorly, but there's something to be said about a tool that is so easy to accidentally run in an unmaintainable way. I also have some limited experience with Drone and Circle, but not enough to talk about them confidently.
Of course GitHub came and blew that out of the water with a composable social actions ecosystem. Which was brilliant and remains the killer feature that puts GitHub far beyond reach of other CI platforms, especially now that they've improved in-repo composability a few months ago.
- Bring your own runner is easiest/most robust
- environment deploys and management
- tons of deployment integrations
- DevSecOps features
- Built in support ticket handling
- Free container & package registry
For most of these I’m pretty sure gitlab had them built into the platform (and free!) first
include: https://gitlab.com/awesome-project/raw/main/.before-script-template.yml
But I agree, the social virality is what makes GitHub Actions what it is.
I used to only visit a GL project when I needed to look through Nvidia cuda container builds… everything else I needed was on GH :shrugs:
GitlabCI's innovation was integration: making it a feature of repo hosting.
When we do use an external dep (like in two places for our massive project) we pin it to a specific GIT sha that we manually curated to not contain anything weird
I find that the only difference is that it's YAML - so even harder to debug, and maintains the same model where you must re run an entire pipeline every commit to test functionality.
Also the ability to rerun, target, and influence builds themselves is better on GitLab as well I think.
I found actions easier to use
IMHO, I don't want bells and whistles from the CI tool. CI tools are dry; I see them as a car drive shaft - it must work properly, and I don't expect it to be pretty. If you need to showcase it, you are probably doing something wrong.
And one more thing: I expect Jenkins to be around and free in the next 10 years. I doubt that will be the case for "insert-your-favorite-ci-from-some-company".
omg switching away from jenkins (in our case to gitlab CI) was a revalation. SO much easier to use. There were a ton of things we were avoiding doing in CI (or at all) that we started doing (easily) once we switched.
> it must work properly, and I don't expect it to be pretty
Too often it really _didn't_ work properly, or the gap between where we were and "working properly" was a mysterious foggy ocean with no clear path.
To be fair, we drove it pretty hard, we had some jobs that ran thousands of tests on big clusters of nodes to validate and deploy huge ETL pipelines, but man it was nice to have that work smoothly with a nice UI that made sense with super-well-documented pipline commands. It did _basically_ work with jenkins, but the experience of troubleshooting problems and adding new features was a constant pain point that really dragged out a lot of work.
When you go beyond that is when everything falls apart. And by fall apart I mean your org starts to grow to a technical ci debt by a thousand cuts until its taking some teams hours to build and ship their code or debug a spaghetti string of code strung together across undocumented boundaries.
Have you used other solutions at all? I used Jenkins for years and a lot of things that that came after that, from industry standards like GitHub Actions and Gitlab CI to more bleeding edge stuff like Earthly and Dagger. Jenkins is the worst of all of them, even if it may do a couple of things “better” than the others
Once you need a tiny bit more, you are screwed. By that time you are already too heavily invested because initial barrier to entry was so low.
Yeah, that's the biggest problem. Almost every thing you need for a CI system comes in the format of a plugin, because Jenkins core is too simple. You can't always say no to your developers' plugin requests, because many of them are actually very reasonable.
Now you install one simple plugin, and it comes with 10 dependency plugins, and that's when your nightmare begins:
- out of these 10 plugins, 4 were owned by some random dudes on the internet and hasn't been updated in 3 years.
- 2 are not compatible with your jenkins core version.
- If you update Jenkins core, 3 other deprecated plugins will stop working.
> IMHO, I don't want bells and whistles from the CI tool.
I can see from the architecture design perspective, Jenkins is simple and boring -- it's a monolithic java app and data is just text files. But I would argue that, its UI is much more complicated and confusing than anything else. You have to install a bunch of plugins to integrate with code repo (github, gitlab, bitbucket...), compute systems (vm, cloud, kubernetes...), you have no choice but install a bunch of plugins, and they changes buttons, menus... I had worked for 3 different companies that used Jenkins, and they felt like 3 completely different websites.
To define a pipeline in Jenkins, your choices are either clickops, or the Groovy big gun. There's nothing in between. I don't have much complain about Groovy itself as a programming language, but if you ask your creative developers to use a programming language to define their CI pipeline, you are gonna see 20 different implementations to detect current git branch name. The biggest Jenkinsfile I have seen has 2000+ lines of business logic. Newer CI products like GitLab CI only allows you to define the pipeline in a yaml file, with a list of very limited directives. That's much more conservative. If needed, one can still run a shell step to capture some weird logic in a script, but that inconvenience is a good enough deterrent.
Tools need to be boring and get out of my way - something "modern" tooling forgets since everyone's paycheck depends on forgetting that.
It has two different UIs (last I checked) which is kind of insane, blue ocean still sucked the last time I used it and the product can’t 100% move away from the old UI (again, last I checked…)
On top of all that it’s not really fun to administer (first-hand experience) where almost every environment will result in plugin hell. I wouldn’t describe the setup process “download a single file and run it” as I look at the documentation, either.
Jenkins can’t give you the “one less tool” advantage that you get with GitHub or Gitlab, either.
Don’t forget that Jenkins’ competition includes free options with mostly high longevity.
That's a good metaphor, and it's significant that there are large innovations happening in car drive shafts as with CI tools:
https://electrek.co/2023/11/29/hyundai-kia-introduce-new-uni...
Drive shaft is a good analogy though because its not simply moving data (unless we’re getting really abstract).
GitHub's being slow? Well that's a GitHub problem. Look, it's been ack'd and they're working on it! Job done.
-
I'm not sure why people are logging into Jenkins and complaining about the UI all the time; that just sounds like your pipeline needs better notifications and people are relying on Jenkins to look at build logs? If you want something prettier to look at just build out a dashboard on Grafana or similar, and centralise your build logs with artifacts and link to them with build failure notifications.
Jenkins is the worst CI/CD tool, except for all the others.
> One of the mostly-false promises of Docker, as it was sold to me by the true believers who introduced me to it, was that, if you do it right, you can run the same docker image, and therefore have basically the same environment in production, in CI, and on your local development machine.
I literally have never worked anywhere that hasn't used the same container in dev/test/prod.
People who have had to deal with processes that rebuilt images for each environment, can you tell me what was different in each image?
It becomes problematic because it then becomes easy for engineers to have a completely different container for dev then is used for prod. I recently found an issue where a dev container was using a completely different base and had a different version of node installed compared to the prod container
If the differences between dev and prod are also programmatic in nature (eg based on flag, and mostly configuration-like values), it should be fine.
However, I think it's less common to run automated testing within the same Docker image you build and deploy to these environments.
This is challenging because you probably don't want to include any build/test related packages in your production image (or layer) but you still want some level of confidence that your CI and Prod environments are the same.
I have often see builds pass automated testing but fail after deployment because our production Dockerfile/layer was missing packages or shared libraries that were present in the CI environment.
When you say "automated tests" do you mean unit or integration tests, or both?
Because unit should generally be in your compilation process, not part of any image, and integration shouldn't require anything different in the image compared to any other environment.
Build it, which includes unit tests, and then deploy and integration tests should be run against your stable API.
Tensorflow used to generate multiple images for different use cases: cpu only, GPU, cpu with Jupyterhub, GPU with Jupyterhub.
So each release would generate minimum 4x container builds.
CPU only images were useful for local testing of ml code at CLI (does it run) when a GPU might no be available.
Jupyterhub variants for simple GUI for development work.
GPU for obvious reasons (run the damn thing).
I can’t remember if they used to do builds for their release against multiple CUDA / CuDNN versions… that might have been PyTorch.
"TeamCity Professional is free – even for commercial use – and has no limitations on features, number of users, or build time. It allows you to configure up to 100 builds and run up to 3 builds in parallel, which is more than enough for most projects"
Yeah, no. There's obviously gonna be a catch in the future.
Initial announcement, 2015: https://blog.jetbrains.com/blog/2015/09/03/introducing-jetbr...
Community anger a couple days later: https://bytecrafter.blogspot.com/2015/09/how-jetbrains-lost-...
Significant changes after hearing the feedback, a couple weeks later: https://blog.jetbrains.com/blog/2015/09/18/final-update-on-t...
I'd trust Jetbrains -- as a company with a few decades of experience serving devs and only devs -- much more than your typical get-rich-quick cloud SaaS startup that's just looking for a quick exit.
Yes, they need income too, but they're one of those few tech companies not aiming for hypergrowth, just a sustained user base.
Even though VScode gets all the love these days, and Atlassian and Github and CircleCI etc. get all the enterprise use, they've made some really cool products of their own too (Space, TeamCity). And their IDEs continue to be excellent.
It's not like TeamCity even has a secret catch... their cloud version DOES have a monthly cost already. It's only the self-hosted version that's free.
I would recommend TeamCity to everyone, it's awesome.
Funnily enough, Jenkins does not have that limitation.
The best way out of this is to subdivide responsibilities:
- Put all build/publish/test logic in Makefiles or scripts in the repo. This means that devs can run it locally as well. The only interaction between a test runner and the codebase should be running `make <target>`.
- Put all permissions and code checkout and artifact publishing credentials in the test runner, but no logic. At most you would put processing of test output like making junit/coverage/tap more web-readable.
That's it. Split the efforts cleanly, and things fall into place. Also, you can switch runners easily - less needs to be reimplemented in whatever runner config DSL is picked.
This is critical!
In many teams Devs are "lazy", if any errors or weirdness happens in CI they just wave their hands and say "not a feature, not my problem". Even though they know exactly what's in the test code, the deploy code, and can see the issue.
Letting Devs see and run the CI scripts locally gives them _ownership_. They don't have to fix the issues, but they have control and understanding which radically reduces the arguments and pipeline drama :)
This always breaks because then each platform has a specific way of defining env vars or secrets for those Makefiles and bash scripts. End result is devs still can't really run CI "proper" the way it's configured in a runner.
The only part of a CI pipeline I can imagine requiring secrets would be a release/publish step. However, those would only occur after the tests run successfully, so nothing up until that point would require secrets.
make <target> THIS=THAT AND_THIS=THAT AND_THAT_THING=SOME_OTHER_THING ETC=ETC
Do you have a better option to suggest?
Your pipelines should mostly be CI provider agnostic and runnable from anywhere, including your laptop during development.
If you ever need to change a CI provider, you just move your pipelines.
I'm definitely biased, I'm working at Garden[0] which allows you to do just that but it's mostly applicable for teams using Kubernetes. I paused at your comment because Garden has sometimes been called "a makefile for the cloud".
Whatever tooling you use, having portable pipelines that you can iterate on and debug from your laptop is the only sane approach imo.
[0] https://docs.garden.io/overview/use-cases#faster-simpler-and...
I think this is a really big scoping mistake. There is a clean glide path, and I have no idea why you'd think you need to replace Jira.
You can use Gitlab Runners in Github (https://docs.gitlab.com/ee/user/project/integrations/github....). Then with your mirrored repo you can try out the Gitlab features and realize it's better to use Gitlab for your repo if you're aso using it for CI. Having switched back and forth over the years (with tens of engineers using the repos), it's really not a lot of work to migrate between Github and Gitlab.
Finally, if you like Jira, just use that? Sure there are advantages to having everything in one place, but if you're already using Jira+Github then you'll get an equivalent experience with Jira+Gitlab.
I think this is one of those things that if you had actually prototyped the migration, you would find that it's much easier than you thought it was going to be.
PMs now complain that the GitLab "issue board" is nowhere near a replacement for Jira and us devs complain that GitHub had a nicer UX and less stability issues.
Can't make everyone happy at once I guess /shrug
Perhaps most critically the interface is just insanely slow! It regularly causes waits in sprint meetings while we wait for someone to drag & drop an issue or for the page to load.
The only reason it's popular is because PMs like to make engineers do their job for them so they can just click a button and get pretty graphs that they can copy & paste to presentations.
Sometimes you don't adopt X because its likely to make people want to adopt Y
I'm currently leaning towards Jenkins, because neighboring teams are using it (with some pain due to their scale, but theirs is 10x-100x ours), and I have some ~10y-old experience with it.
(Our builds are compiling for embedded targets, with a Makefile-based system)
[1] https://woodpecker-ci.org/
[2] https://social.treehouse.systems/@marcan/111258515328251078
GitLab's not too bad. I had a Gradle pile I had been using but once I knocked out some old Java bits I wanted to sleek my "required tools" down a bit. I've integrated some OpenProject stuff into GL also. Maybe someday . . for "minimal" clients . . I can just bring in the GitLab and call it a day, so far as server stuff is concerned.
That day is not today, however. Customers in my business want a lot - like, a LOT a lot - of project toys, and homebrewing those in GitLab is beyond my skills.
I think though ironically my take away after setting everything up, was that I didn't really need to spend a lot of time to get it perfect, rather just getting it working was enough for me.
Unclear whether that’s enough for your situation but it’s by far my favourite because of how it lets you store the CI pipeline in the repository (so changes are reviewed via normal channels etc).
Edit, forgot to add that I recently interviewed a bunch of candidates for a high paying job at a ai hardware company and every one of those candidates stated they worked on or near a project that involved moving from Jenkins to Gitlab CI.
I disagree with this pretty strongly. I've worked with Jenkins where it was all freestyle jobs and it was a nightmare to maintain. Pipelines written in groovy are essential to doing Jenkins well, imo. It's also not exactly hard to write groovy. It's basically Java, not some weird esoteric language nobody knows.
Theres also the limitation that functions can be max 1000 bytes or whatever I don’t remember. Managing shared libs and testing them is not easy.
It’s not basically Java at all. Everything is subtly different and will surprise you in the least convenient moment.
I guess I'm weird then, having produced about 100K LoC in Groovy over the last couple of years.
We'd move to Gitlab CI too — if it was anywhere near resembling a serious tool and not a sandbox toy collection.
No it dosen't or maybe it does for java which I don't care about.
The ui looks like from the 90s.
Finding anything in bigger projects/companies is impossible... I have browser bookmarks to find anything.
The log view is annoying at best.
I have nothing good to say about jenkins, it feels like it is stuck in the 2000s and refuses to change.
I've used Buildbot, Travis, Gitlab, GitHub and Teamcity. None of them are anywhere near as bad as Jenkins. Even Buildbot has a much much saner UI and that's a weird Python CI system that practically nobody uses.
Jenkins is just an objectively terrible choice.
Is the Space Jam website peak UI design? What about Microsoft’s “hell of tabs” settings dialog boxes?
Or the original Amazon home page: https://www.versionmuseum.com/images/websites/amazon-website...
What link do I click to find my order status? Where do I go to search for a book by keyword? What will happen if I click the link for “first time customers click here?”
Same deal with Yahoo!, what to the “new” and “cool” and “more yahoos” buttons do? http://2.bp.blogspot.com/-JAp1M-Q_0_Y/TkAwsyueWxI/AAAAAAAAAM...
If I want to check a build status on my phone what’s the excuse for the web page to not display nicely on that device? You really think there’s no good reason for me to want to do that? Doesn’t modern CSS allow you to completely customize the viewing experience for both platforms in the same codebase?
You often end up with these glorious Rube Goldberg machines where Jenkins does everything and becomes a single point of failure - if the instance falls over, everything’s fucked.
My issues with Jenkins:
1. Groovy is a really annoying language to configure stuff in. I’ve never written it for other things but it has so many little idiosyncrasies that catch you out.
2. Jenkins doesn’t have much in the way of resource management. It’s easy for a job to take over the entire node and Jenkins doesn’t have the smarts out of the box to monitor this and put things in a different node.
3. So many random job killing bugs that boil down to poor job isolation and resource tracking. Every now and then killing one job can take down another.
4. The UI and UX is bad. I know there are better interfaces for it, but they all just have such limited interface exposure of state beyond “running” and “failed”. Estimate time is useless because it aggregates failed jobs.
5. Inter job dependency management and tracking is poor and annoying to setup. It’s doable, but really should be a better first class citizen.
In many ways, I start looking to other tools to fill my CI need that aren’t meant for CI: Render farm managers.
Film farm rendering is largely similar to CI at a structural level. But they have much higher throughput of individual jobs and have built around making the most of that. There’s better inter job dependency tracking and resource management because of that, and they’re built to expose more information up front to help non-tech savvy artists or render wranglers who need to see status quickly at a glance on a few hundred jobs at once.
Great little language though, IMHO one of the best for quick prototyping.
But things like the differences between single quotes and double quotes and a bunch of other subtleties like that, make it really frustrating when the only time I use it is updating build logic.
I’d honestly prefer a more consistent language like Python or Lua. Even if I don’t code them all day, they have more resources to lookup and fewer gotchas.
That's the thing I hate that about CircleCI, Github Actions, etc... having to "commit, upload, wait" to test every minor change =/
If done correctly, the Jenkinsfile should be easy to read by a person who has never written a Jenkins pipeline before.
With this approach it is hard to mess things up by creating a monster pipeline which no one can understand, and if necessary, you can quickly change to different CI runner at a later point in time.
this lands, and not specifically for jenkins. CI generally is inscrutable and difficult to test
every CI has its quirk
github CI feels fundamentally worse than gitlab because job artifacts are native in gitlab -- this makes a million things easier. github's own action for attaching things to releases is deprecated. the upload-artifact action doesn't work in nektos/act (the only way to test locally)
Jenkins is a bit like Perl IMO: there are ways to use it that avoid the majority of its sharp edges (eg putting logic in bash and just call bash, avoid groovy) if you know the perils that lurk.
You can also run a "less privileged" container with all the features of Docker by using rootless buildkit in Kubernetes. Here are some examples:
https://github.com/moby/buildkit/tree/master/examples/kubern...
https://github.com/moby/buildkit/blob/master/examples/kubern...
It's also possible to run dedicated buildkitd workers and connect to them remotely.
I don't know how that compares to other solutions like github though. Being able to run a really lightweight container in argowf has its advantages
But the problem with Jenkins is the problem with all CI. Our (software in general, not Jenkins specifically) build process are all godforsaken nightmares that would put HP Lovecraft and Rube Goldberg into rubber rooms. You can't add a bunch of extra automation to any of these systems and get something better, let alone a system that needs to be able to automate so many different and unique snowflake horrors.
They already include test frameworks.
It would be cool to have, at least as an experiment, a language with no snowflake horrors at all, if it runs at all, it's in the language or framework's One True Way All Tools Understand. Static typing, maybe even linter warnings as fatal errors, anything that makes adding CI a hassle just doesn't exist.
Maybe it could even refuse to run without full test coverage.
So you could take any project at all, click "Turn on CI" and that's that, there's only one CI config that exists.
I'm guessing the number of use cases would be tiny to nonexistant in the startup centric tech world, but they might be some of the least stressful projects ever made!
It requires an iron clad vision, willingness to be a pariah among the coding masses and a dictator internally, excellent communication in order to present the vision and progress to the outsiders who are interested, and piles and piles of money to pay for it for years before it becomes sustainable, however you plan to make it so.
Richard was my manager in early 2019, and was awesome.
What happens when the CI team says no and it introduces complexity in your system? Or you end up doing “shadow CI” on your machines to avoid conflict and escalation?
The complexity has to live somewhere. It is always worth evaluating the optimum place for your organization at any given stage.
The whole post can be summed up as he had very little CICD experience. Made lots of beginner mistakes, which is easy to do in Jenkins. Then decided to write a post where all his complaints about Jenkins are not only wrong but are the issues that plague all the other CICD tools.
> So instead of writing Bash directly, you’re writing Bash inside Groovy
Why are you doing that? You have a fully featured programming language and you are running `sh('npm install')`. You could do this instead https://github.com/DontShaveTheYak/jenkins-std-lib/blob/mast... . How is bash inside of YAML better?
> The trouble is: Groovy is a much, much worse language for executing commands than Bash. Bash is interpreted, has a REPL that is great for experimentation, does require a ton of imports, and has lightweight syntax. Groovy has none of these things.
Groovy has a language server, linters and a vscode IDE plugins. They are probably not as stable or full featured as the bash ones, but they are available and very few take advantage of them. Again, how is YAML+Bash better?
> The way that developers test their Groovy steps is by triggering a job on the remote Jenkins server to run them. The feedback loop is 2 orders of magnitude slower than it is for just executing Bash locally.
This is a rookie mistake. For about 60-75% of pipelines you can run them locally in a docker container on your local machine. You can even set up hot code reload so as you change your pipeline the Jenkins reloads it. You can also configure the job to kick off a build when it reloads the code. When Jenkins is configured correctly it has the fastest feedback loop of any CICD tool on the market. GitHub actions comes in a close second since it can also be run locally but you cant run a "clone" of what you run in production, like having the same secrets, so it gets second place. Beside Jenkins and GitHub actions, I dont know of any solutions for the other tools.
You can run a GitHub action on Jenkins. It's a very deep and complex system. It's like an iceberg and so many engineers dont leave the surface before deciding it sucks and one of the YAML CICD tools is better. Sure the YAML alternatives are EASY to get started with and to do basic stuff with. But they are Terrible at anything complex. While Jenkins is not easy to get started with, once mastered, you can build complex pipelines with ease.
I get that I'm a Jenkins fanboy. Most of the things I mentioned above, I either contribute to or I'm the author of. I know Jenkins has issues. I know it has hurt lots of people, I read the complaints online. But it's still the best out there. The best software in the world is not written in bash or yaml and the same is true of the best CICD pipelines in the world. It's a shame very few people get to see/use those pipelines.
You shouldn't judge a developer tool by just what is possible to do with the tool. After all, with a little turing-completeness it is possible to do anything with anything -- you should judge the tool by what is easy to do with the tool. A good developer tool shouldn't require knowledge of a bunch of arcana to "configure correctly". A good tool protects you from "rookie mistakes" and makes sane choices the intuitive and obvious path of least resistance. Good tools can have a learning curve, but they assist the learning curve by making their abilities easy to discover and experiment with, they don't require you to dig into source code or do random searches on github to find some random pipeline somewhere that uses the configuration you need, as described in the post.
I'm judging the article for being incorrect about it's specific points and for focusing on a single tool while the alternatives also suffer from the exact same issues.
There are always issues, and like the author, it's "only" about 25% of the time. A quarter of all builds fail due to things absolutely not related to that specific build.
> oh, we were resource constraint at that one second, retry and it'll probably work
> oh, we can't replicate that error message from a tool that you have no visibility over, retry and it'll probably work
> oh, we have no idea why something timed out, retry and it will probably work
Now when I have a failed build I don't actually trust the build before I don't trust that there was an edge case in an integration test that failed. Because it's almost always something outside of the code.
[Edit] downvotes but no constructive responses? The issue isn't necessarily Jenkins, it's the type of people who use Jenkins to roll their own everything.
Had the same issues with (self-managed) gitlab CI. This is not jenkins-specific and usually something you can fix much easier in jenkins (other than just retrying)
> oh, we can't replicate that error message from a tool that you have no visibility over, retry and it'll probably work
This could maybe be better in other, more declarative CIs but honestly, its pretty much the same. The only thing some of them have is a local runner.
> oh, we have no idea why something timed out, retry and it will probably work
Jenkins has this, yes. But use another CI with docker and you can get the same and much worse.
But in essence, yes, at our company too, jenkins is a pet. But this means that someone knows it from the inside and outside and knows which things to fix in your pipeline. Good luck with trying to extend gitlab CI (or others) to fix things you need and/or opening an issue with them.
Gitlab has quite some high-profile issues sitting around for years with customer requests piling up. In jenkins, you'll find an extension that will do exactly what you want, or you could roll your own.
I didn't say this was exclusive to Jenkins, just that Jenkins has always been a subset of this problem. Selfhosting this, and wanting to make it completely customised to fit the whim of whatever someone in compliance/security/accounting decided is an issue. It doesn't help that Blue Ocean is a terrible UI, but that's just an ugly facade to a system that enables bad habits. Other systems can be abused, but whenever Jenkins appears it is always abused.
If you're doing something that doesn't line up with Pivotal / Cloud Foundry workflows, you're probably going to have a bad time.
I put, however, that only companies have the problem, not programmers, and not even regular (sized) engineering teams; except when they have a fast employee turnover speed.
I do not have a very clear idea, but roughly speaking I'm saying that java does really well in the "too many engineers are working on the same code but not at the same time" problem space
More often than not, I end up creating a sandbox folder in Jenkins and cloning workflows to work on them by hand without having 90 commits of misc errors and typos. This has helped speed up some iteration.
Sort of off topic from the article, I think where almost every single CI engine falls short is the extremely limited scope they cater to - that every job ever made must be solved with the cargo cult ideology where every CI job mustn’t ever have human input, everything must be perfectly automated CI. This is wonderful when it is possible to do, but sucks when it isn’t feasible.
Jenkins’ strength is that it’s actually a workflow engine masquerading as a CI tool, and that gives a lot of flexibility for a number of bespoke processes.
Do you want to wait for someone to click a button to grant permission to run a job because the test might spin up extremely expensive resources and you’d really rather someone baby sit it? Jenkins has you covered.
Or maybe it just isn’t possible (or worth the time) for your team at a point in time to automate all input to a job, so spin up a Jenkins job that lets someone manually specify parameters.
Maybe your flow can trigger a release of software but you want a human to come behind to do a manual once-over as a final check, then also capture who/why approved it.
More CI tools need to accept the reality that not everyone can implement the perfect ideal of CI jobs. Some jobs will require a human step somewhere, be it custom input to a job or waiting for a response.
Dagger is the answer to jenkins woes. It's saved my sanity and made CI development tolerable again.
* Real language support. No groovy, no yaml.
* Debugging available in your native IDE tools.
* Clear docs in any language of your choice.
* Reproducible builds you can run locally. Just need docker.
This all comes with the bonus that it can actually just consume GHA yaml with some of the tooling folks have made.
I wouldn't recommend it for general CI other than that. Think of it like a butler, able to do small jobs, not as your complete home assistant.
And, of course, everybody should be using whatever language and version they want to, and run their software on whatever cloud they want. The notion of this makes me super uneasy, but at the same time it's a good explanation why many big orgs have 100s of engineers when they can probably easily do with 10s of them.
Every job should be a few simple commands or a python script or something I can run locally. This simplifies reproducing and debugging issues caught by CI -- there's nothing worse than wasting time trying to reproduce what happened.
The discoverability issue is also a massive issue, declarative pipelines massively over promise its capability. Pipeline configuration was mostly still done in the UI because extensions had 0 documentation about how to declaratively define its configuration. Searching on the internet for the magic incantations as the author puts it is such a waste of time and I just compromised to get it running.
I'm glad this post is out there to articulate these common challenges when you invest so much into Jenkins.
I use the same Dockerfile for production builds and for running all the CI related tasks, without any problems. And the same single pipeline is used for PR hooks and releases.
Post also mentions Blue Ocean UI, but that one is deprecated and should not be used at all.
Jenkins is far from perfect, but unless you can use Gitlab (and sadly you can not in every case), it is the most feature rich self-hosted tool out there. Just make sure you set it up with Jenkins Configuration as a Code plugin out of the box.
I wish there was a nicer tool that worked. But there isn't.
* This, of course, applies to all long commutes everywhere, like how it applies not only to Jenkins, but lots of other software too… especially Jira.
The worst thing about Jenkins is that it works. It can meet your needs. With a liiittle more effort, or by adopting sliiiightly lower standards, or with a liiiiitle more tolerance for pain, you can always get Jenkins to do aaaaalmost what you want it to.
And this is what keeps Jenkins afloat, despite it being a consistent source of CVEs both in core and in its myriad of plugins that don't bother to follow semantic versioning so every upgrade is playing Russian roulette with your company.
I guess the flexibility with jenkins is probably both a blessing and a curse.
i’m delighted to pay 30% over infra cost for convenience, but not 500%. and it better actually be convenient, not just have a good landing page and sales team.
this month i learned localzones have even better spot prices. losangeles-1 is half the spot price of us-west-2.
for a runner, do something like this, but react to an http call instead of a s3 put[1].
for a web ui do something like this[2].
s3, lambda, and ec2 spot are a perfect fit for cicd and a lot more.
1. https://github.com/nathants/libaws/tree/91b1c27fc947e067ed46...
2. https://github.com/nathants/aws-exec/tree/e68769126b5aae0e35...
I had a good chuckle at this.
sh 'make all'
sh 'make test'
Well there's your problem right there.
using jenkins declarative pipeline, almost everything can explain in code.
updating jenkins is a walk in the past.
the most tedious part is dealing with no longer maitenance or support plugin.
make sure to never expose jenkins out. use a relayer to bring webhook traffic in
It's fast, simple and took a couple hours to set up. Rarely ever needs attention. It just works.