That argument always struck me as vacuous. Dump a barrel of ball bearings on the top of a craggy hill. Wait as they all bounce around, some getting stuck in local minima and some bouncing over obstacles and covering large distances.
Would you claim that they all traveled the same distance because they all traveled for the same amount of time?
Evolutionary space is very high dimension, which makes the argument that just projecting onto the (1d) time axis is misleading even stronger.