Is it? I've written hundreds of thousands of lines of production Rust, and I've only sparingly used unsafe. It's more common in some domains than others, but the observed trend I've seen is for people to aggressively encapsulate unsafe code.
Unsafe Rust is quite difficult to write correctly. (The &mut provenance rules are a bit scary!) But once a safe abstraction has been built around it and the unsafe code has passed Miri, in practice I've seen people be able to not worry about it any more.
By the way I maintain cargo-nextest, and we've added support for Miri to make its runs many times faster [1]. So I'm doing my part here!