The "don't touch it if it's not broken" philosophy is fundamentally at odds with an internet-connected machine.
You either need to upgrade or unplug (from the internet).
There are still places out there that are running WindowsNT or DOS even. Because they have applications which simply won't run anywhere else or need to talk to ancient hardware that runs over a parallel port or some weird crap like that. These machines will literally run forever, but you wouldn't connect it to the internet. Your hypothetical cave device would be the same.
Upgrading your OS always carries risk. Whether it's a single yum command or copying your entire app to a new OS.
Besides, if you're on CentOS 8 then wouldn't you also be looking at Docker or something? Isn't this a solved problem?