The compiler is a huge assemblage of C++ templates, stitched together with a little Python.
The generated code is in an assembly language for a highly parallel machine, and needs to be heavily optimized down to the cycle.
But the compiler itself does NOT need to be so minutely tuned. It needs to be maintainable and it needs to allow easy development of complex code generators.
Go would be a great choice for the compiler.
Instead, we struggle with C++ templates, CMake'ing our code slowly and with absurd complexity, modifying the compiler slowly and with great difficulty...
Also, you should not discount compiler speed completely. You, yourself, were upset at the speed of a compile process. All of your users are going to be waiting to get things done while the compiler is running, so you might want to do some math as to how much a 10% slowdown would cost in developer salaries. The makers of gcc, LLVM, and clang spend a lot of effort on improving the speed of their compiler to make sure that you have minimal time waste.
Get a handle on your code quality, and put the language features to work to solve problems for you. Then Go will increasingly look pitifully inadequate.