You are right. I have no experience with Go, but that's a fast compiler for sure.
Though they miss out heavily in the typing strength. If only it had Result/Either/Maybe types in the std lib from day 1, and proper sum types with some pattern matching switch statement, that would have made the language soooooo much better.
I find Elm also has good compile times. Maybe polymorphism and/or type-classes are what is hard to optimize.