There are definitely trade offs when switching from Elixir or Erlang to Go. If you're a functional programmer who can't live without immutability, or you plan on running a cluster of machines that can communicate with each other and hot-swap code into the running system, then Elixir and Erlang are good choices.
If you have some extremely CPU intensive code, or you like cross compiling to a lightweight binary, or you need static typing (without giving up preemptive scheduling), Go is a decent choice.
Elixir and Erlang are not slow by any metric, but there are faster languages. Discord famously had to augment their Elixir code with Rust for example to scale to 11 million users [1].
[1] https://discord.com/blog/using-rust-to-scale-elixir-for-11-m...