When it's in the faraway part of its orbit, it is moving very slowly, probably only tens of meters per second, but it's still close enough to the sun to eventually fall back in for another loop.
However, if something else dense enough got close enough out there, it would be easily perturbed and have its whole orbit altered, or even be ejected.
But interstellar space is pretty void of wandering solid bodies, so it keeps falling back towards the sun.
