The type system is probably not the bottleneck in any of those cases. As a sibling comment points out, ocaml has very good compile times, and the inference problem is basically the same as in Haskell.
In the case of rust, I suspect one of the biggest issues is the way parametric polymorphism is implemented. Basically, if in your program you end up using e.g. Box<usize>, Box<MyType> and Box<Result<String>>, you're compiling Box 3 times.
I don't know enough about swift to hazard a guess as to where the build is spending its time.
My experience with Haskell is that compile times are neither great nor terrible.
