The analogy fails there because our 3manifold space isn't necessarily "embedded" into a higher dimensional construct the way the surface of a sphere is embedded into our 3 dimensional universe. So there isn't necessarily a hyperdimensional up or down to consider.
Or, alternatively, if you think of time as a dimension, then we can call "up" the future and "down" the past. In that case, if you look down inside the past, you'll see that in that analogy the center of the universe corresponds with the big bang! And all the galaxies are equidistant from that point/moment in spacetime.
