It's crucial, for sure. But there are trades for which it is the only determinant, and those are dangerous waters.
There are market participants who make their money on pure latency arbitrage; it is a 100% speed game (because anyone can figure out that 1 - 1 = 0).
The majority of HFT falls into the "you have to be fast enough to not get run over" bucket.