The example of Paperclip Maximizer was meant to be read as:
template<typename Paperclip>
class Maximizer<Paperclip> { ... };
Powerful optimizers aren't science fiction. Life is one, for example.
"Fundamentals of ML" were known by these people. They also don't apply to this topic in any useful fashion.
Exponential growth of that form that is totally unchecked simply does not exist. All things have limits that check their growth, and the assumption a computer will grow like mad is exactly that - an assumption - one formed from extreme ignorance of what general AI will look like.
In Earths history we've had any number of these "uh oh" events. The Great Oxygenation Event being one of the longest and largest. Little tiny oxygen producing bacteria would quickly grow, and the check that stopped their growth was a massive set of free radical death that killed nearly every living cell in the ocean at the time. And then the system would build back and do it again and again.
Ignoring black swan events, especially of your own making, is not a great way to continue existing. If there is even a low chance of something causing an an extinction level event, ensuring that you do not trigger it is paramount. Humans are already failing this test with CO2 going "Oh, its just measured in parts per million", not realizing that it doesn't take much to affect the biosphere in unfriendly ways for continued human existence.
DNA/RNA is a strong version, but luckily we're aligned with it by evolution, but still at risk of extinction by disease.
The fact that corporations try to improve themselves and innovate with time has nothing to do with AI or LLMs and their risks.
The fact that evolution exists also has nothing to do with it.
I am not able to rightly apprehend the kind of confusion of ideas that could provoke such a statement.