We do the pre-baked dependency images too, and it's definitely workable, but I feel like it's a lot of overhead maintaining those— you have to build and distribute and lifecycle them, and it's extra jobs to monitor. Plus you now have an implicit dependency between jobs that adds complication to black-start scenarios. I wish tools like GitLab CI had more automated workflows for being able to automatically manage those intermediate containers, eg:
- Here's a setup stage, the resulting state of which is to be saved as a container image and used as the starting point for any follow-on stages that declare `image: <project/job/whatever>`
- Various circumstances should trigger me to be rebuilt: weekly on Saturday night, whenever repo X has a new tag created, whenever I'm manually run, whenever a special parallel "check" stage takes more than X minutes to complete, etc.
Ultimately, I think the necessity for all this kind of thing really just exposes how weak the layered container image model is— something like Nixery that can delivery ad-hoc environments in a truly composable way is ultimately a much better fit for this type of use-case, but it has its own issues with maturity.
