That's true, but again, it's also true in most other languages that your dependencies can do anything they like, including shipping untrusted, compiled C code.
The value of something like "unsafe" is not that your code is magically protected if it isn't there, but rather that it provides a warning sign as to where dangerous code might lie. So if something goes go wrong - or you're just worried something _might_ go wrong - you know roughly where to look. In the same way, if your Python code includes a C extension and you start getting weird segfaults, you can reasonably guess where to start looking.
Except that, because unsafe is so well integrated in the language, you can often significantly reduce the impact of unsafe to only a few lines, and in most cases you won't need to use it at all.