Security teams are often staffed by people who have no operational experience, and do not understand the consequences of what they are recommending or even mandating. Often those staff are blindly following hardening guides or asking for every configuration switch to be flipped to "most secure" setting without having a good understanding of the threat model for the workload and without taking into account the tradeoffs between utility and operational cost. The level of advice can be on par with ChatGPT or worse, but it is taken more seriously due to the advice-giver's job title.

Security teams often have no "skin in the game". There are no real disincentives to stop them from asking for crazy or very expensive things and imposing high costs on other teams. In fact they are incentivised to do that very thing, because the only thing that covers your butt more than recommending everything possible, is recommending everything possible PLUS some things that can't be done with the time & budget available, leaving them able to say "we see you had $security_problem, well, we recommended $impossible_thing but you didn't do it" (e.g. say, $500k of DLP solution [with its own operational risks!] for a workload that only makes $1M a year). To be fair I've seen some good & practical security teams but once you get a bad actor / games player at management level that behaviour can become very sticky.

I have seen variants of this nearly everywhere I've worked, it seems very hard to get incentives aligned between the do-ers and the secure-ers.

The most practical workaround I've seen is to make sure there is a reasonable balance of political power between the various parties.

"Reasonable" can be hard to establish but is context dependent. You would expect "Security" to have more power in an F500 because the brand value, financial and legal exposure are high and individual dev teams aren't necessarily across or exposed to the full consequences of the damage they can cause.

In a startup you would expect "Security" to have much less power because the consequences of not shipping / misallocating effort are almost immediately existential.

1. put all assets into infrastructure you control, allow pulling only from that infrastructure, no exceptions.

2. understand your containers enough to be able to modify and manage their trusted CA / TLS chains.

You really ought to be in a position to manage your supply chain and you really ought to be in a position to manage what your containers trust. Certainly more work and inconvenient, and maybe not practical everywhere, but worth doing even independently of a middleware box, at least once you get to some level of "in production for reals"