If something is defined or not is a property of the language specification, as you stated.
What you’re now bringing up is something different: should a specification define this behavior, or not? I think you’ve properly identified a trade off, but mid-identified the details. Defining a behavior here does privilege certain architectures, but it doesn’t make it impossible. It means the behavior must be replicated in code on some architectures, which is slower.
This is the trade off that Rust (and apparently Zig) are making. This is informed by the architectures that exist, and which ones are wished to support. At this point, two’s compliment is overwhelmingly used, and so the decision was made that defined behavior is worth it. Note the parallel here with POSIX specifying that CHAR_BIT must be set to 8.
Notably, the situation here is so overwhelming that even C++ is specifying two’s compliment: C++20 says that signed integers must be. This was recently voted in. I haven’t read the paper recently and I don’t remember if it changes the UB rule here or not, but it does define the representation of integers, a very similar thing that’s the motivation for UB here.