The optimisation passes are expensive (not the largest source of compile time duration though).
Debug mode is designed to build as-fast-as-possible while still being correct, so that you can run your binary (with debug symbols) ASAP.
Overflow checks are present even in release mode, and some write-ups seem to indicate they have less overhead than you’d think.
Rust lets your configure your cargo configs to apply some optimisation passes even in debug, if you wish. There’s also a config to have your dependencies optimised (even in debug) if you want. The Bevy tutorial walks through doing this, as a concrete example.