For simulations and scientific calculations, I do agree, to a vast extent. But in a world that is moving more and more towards zero-trust networking, even many of those will start being looked at as potential attack vectors into other systems.
You see, it is absolutely expected and required that our applications will load and run arbitrary 3rd party code, generally with the expectation that it lives in the same address space as our application (though this is not formally required).
No sockets, no network, no backdoor hacks. You write code, call it a VST plugin, make it sound desirable ... we are expected to load and run it.
Yes, several DAWs have made the move toward out-of-process execution of plugins, but that doesn't begin to address the myriad problems caused by loosely-written plugin APIs not adequately pinning down threading, thread priority, memory access and more.
Filesystem access? Of course! That code runs as you! Because you want it to!
This capability exists in completely open source, such as OpenZiti - https://openziti.io/.
If you care about program correctness in any real sense, memory safety is table stakes.