Yeah, I was researching further and found this[0] SE question. Seems to also say the same thing that there is a lot of "space dust" between here and there.

[0]: https://astronomy.stackexchange.com/questions/30339/why-not-...