Fast (and principled) inference for not-too-complicated polymorphic types is an explicit design goal for OCaml and Haskell languages. And this design goal constraint quite a lot the type system.
Typically, there are many type system features that break this property and where OCaml and Haskell requires type annotations (for instance polymorphic functions that takes polymorphic functions as arguments and use them in a polymorphic way).
As a consequence, typechecking for OCaml programs tend to take from 10% to 60% of the whole compilation time for typical programs. However, for OCaml programs that make heavy use of GADTs typechecking can dominate the compilation time (probably due to the exhaustiveness check but I have yet to empirically check that).
Infamously, Swift made the choice to introduce a typechecking algorithm with an exponential complexity in the size of expression (to support function overloading and literals) even for simple types.