Because the rumor mill loves to churn things up, and people forget the past.
The original iPod was an incredibly niche product. It required a Mac at a time when Macs were way less common than they are now. When Windows support was added, it required FireWire, which was quite uncommon.
The original iPhone did OK but didn't sell super well. It was very expensive, had only 2G connectivity when 3G had already arrived, only worked on the #3 cellular carrier, and didn't support any third-party apps.
The original Apple Watch was bulky and severely underpowered.
Apple continued to iterate on all of these and they ended up being quite successful.
That's not to say Vision Pro will see the same treatment. There are plenty of failed products you can point to as well. Just that an iffy initial release doesn't mean anything about the long-term outlook for the product.