rustc's trend the last twenty releases has been that the compiler is getting faster. I guess every dependency is slowly growing in size, offsetting this in your perception? The compiler benchmarks are of course held constant to be able to compare releases.
See the perf website https://perf.rust-lang.org/dashboard.html