Yes, that is the least scary. If species like us are stuck in our own solar system, we're eventually doomed. If we're not, it would only take a few million years to colonize the galaxy. If two non-stuck colonizing species exist at the same time they have to arise within a few million years of each other, in a galaxy thirteen billion years old.

Frankly, us being the only/first sapient species is by far the most optimistic possibility.