I don't think it's quite as easy to guarantee panic freedom as you think.
For example: do logging frameworks guarantee no-panic behavior? People can add logging statements practically anywhere, especially in a large team that maintains a codebase over significant time. One innocuous-looking debug log added to a section of code that's temporarily violated invariants can end up putting the whole program into a state, post-panic, in which those invariants no longer hold.
A lot of experience tells us that this happens in practice in C++, Java, Python, and other excpeption-ful languages. Maybe it happens less in Rust, but I'd be shocked if this class of bugs were absent.
Note that I'm talking about both safe and unsafe code. A safe section of code that panics unexpectedly might preserve memory safety invariants but hork the higher-level logical invariants of your application. You can end up with security vulnerabilities this way too.
Imagine an attacker who can force a panic in a network service, aborting his request but not killing the server, such that the panic on his next request grants him some kind of access he shouldn't have had due to the panic leaving the program in a bad state.
I'm not seeing Rust people take this problem as seriously as I think is warranted.