I don't doubt that's often true but many Rust projects may be outliers here. A full, non-incremental build of a Rust project involves building all of its dependencies. This can add significant amounts of time if a project uses a big framework like Actix-web, which adds many dependencies.
My tests however run very quickly, ~1ms each. So running thousands of tests only takes a few seconds, even on relatively slow gitlab runners.