But using ordinary module encapsulation and private fields, you can scope the code that needs to uphold those preconditions to a particular module.
So the "trusted computing base" for the unsafe code can still be scoped and limited, allowing you to reduce the amount of code you need to audit and be particularly careful about for upholding safety guarantees.
Basically, when writing unsafe code, the actual unsafe operations are scoped to only the unsafe blocks, and they have preconditions that you need to scope to a particular module boundary to ensure that there's a limited amount of code that needs to be audited to ensure it upholds all of the safety invariants.
Ralf Jung has written a number of good papers and blog posts on this topic.