Unsafe is perhaps poorly named, and several Rust core team members have commented as such. It doesn't mean memory unsafe, it means "not checked by the compiler".
Safe APIs that contain unsafe blocks must still be proven correct, via Miri, a model checker, formal proof, etc. Any safe functions that violate memory safety are considered bugs. The limited number of unsafe functions exist as helpers to build safe APIs when the compiler's borrower checker is insufficient.
What this means is that to verify memory safety, one can restrict their search to unsafe blocks. And hypothetically if the Rust compiler were to get much smarter, it should be possible to prove to the compiler that those blocks are safe (via theorem prover, perhaps?) and remove the "unsafe" declaration.
In most languages, there is no such distinction between the "memory safe" common set users ought to use and the subset that has to be verified independently. Neither Zig, C, C++, nor even Go have a clear delineation between safe and unsafe code.