undefined | Better HN

0 pointsmaratc3mo ago0 comments

My issue with GHA and other "dead simple" systems is that my CI is complicated. Having a real programming language for stuff like "calculate what date it was a week ago" or "concatenate these three strings but only under some conditions" or "parse the output and build an object out of it" is really helpful while a bastardised YAML-based Jinja template simply can't hold up.

But yeah, if all there is to do is "git clone && build && deploy" then Jenkins is an overkill and it probably wasn't warranted in the first place.

0 comments

0xbadcafebee3mo ago

For complex logic I don't rely on the CI system; I've been burned too many times. I shell out to an external program and have it return an output variable, and I just do "if $foo = y then blah" in the CI's DSL (and I keep those tests to a minimum; rather have more separate jobs than one complex job). Often I will put everything in a dedicated build tool (Make or similar) so I can run it from my laptop or CI, and any change to logic only happens in one place. It's adding an abstraction, but the end result is I write the CI job once and never touch it again. For flexibility I add parameters to the CI job.

maratcOP3mo ago

> For complex logic I don't rely on the CI system

I do that too, until my complex logic belongs to the CI system and nowhere else.

As an example:

    It's 2 am now and Jenkins needs to run some (but not all) nightly tests[0]. To figure out which, let's bring the source code and analyse the configuration file, disregarding anything that doesn't need to run at this hour. Once we have the plan for which tests to run, let's figure out what we need to build[1]. Also, let's see what is the status of the pool running the tests[2] so we can decide on a tests parallelisation strategy for this run. When we have a plan, let's build and test, keeping an eye on the triggered tests. When all these have finished, analyse the failures[3], create tickets for the failures[4] and prepare a report to be sent.

I wouldn't be able to express all this in YAML.

[0] other "nightly tests" run at 1, 3, 4 etc.

[1] this is mapped in the configuration file too.

[2] this is internal to Jenkins

[3] same

[4] this involves finding the "responsible person," so a lot of API calls

0xbadcafebee3mo ago

Sounds like you're using cron as a complex job queue! A lot of teams get there eventually, and either 1) keep hacking on cron/jenkins/etc to make this work, 2) invent their own queueing tool (NIH syndrome; been done many times before, there is nothing new to make here), or 3) use a purpose-built solution for this. Airflow is the old-and-busted solution; the new hotness is newer generations of the same concept (Prefect/Dagster, Luigi, Temporal). But often sticking to your existing thing is cheaper; depends how much custom engineering you want to invest.

Fwiw, I do believe you can do this in GHA, but you may need to call their API from your workflow. In addition, their replacement for Groovy is to run an action which lets you embed Javascript/Typescript and call their SDK. It sucks, but so does Groovy! ;-)

j / k navigate · click thread line to collapse

0 comments

0xbadcafebee3mo ago

maratcOP3mo ago

> For complex logic I don't rely on the CI system

I do that too, until my complex logic belongs to the CI system and nowhere else.

As an example:

    It's 2 am now and Jenkins needs to run some (but not all) nightly tests[0]. To figure out which, let's bring the source code and analyse the configuration file, disregarding anything that doesn't need to run at this hour. Once we have the plan for which tests to run, let's figure out what we need to build[1]. Also, let's see what is the status of the pool running the tests[2] so we can decide on a tests parallelisation strategy for this run. When we have a plan, let's build and test, keeping an eye on the triggered tests. When all these have finished, analyse the failures[3], create tickets for the failures[4] and prepare a report to be sent.

I wouldn't be able to express all this in YAML.

[0] other "nightly tests" run at 1, 3, 4 etc.

[1] this is mapped in the configuration file too.

[2] this is internal to Jenkins

[3] same

[4] this involves finding the "responsible person," so a lot of API calls

0xbadcafebee3mo ago

j / k navigate · click thread line to collapse