Luminosity of the sun: ~380 yottawatts (3.8 * 10^26 watts)
Sunlight conversion efficiency of a silicon based solar panel: ~20%
A Dyson swarm around the sun built with silicon solar panels: ~76 yottawatts
A Dyson swarm around the sun where 5% of the panels work: ~3.8 yottawatts
The leftovers from a Dyson swarm around the sun where 5% of the panels work and 89% of the output has been used for interplanetary yachts: ~418 zetawatts (4.18 * 10^23 watts)
Primary power production on Earth: ~20 terawatts (2 * 10^13 watts)
10000 years times 20 terawatts is 10000 * 365.25 * 24 * 60 * 60 * 10^12 = 3.16 * 10^24 joules
Since a joule is just a watt-second, it takes 7.6 seconds for that 418 zetawatts of leftover Dyson swarm output to match up to 10,000 years of current human energy consumption.
or
you're saying that if we can harness that much energy amount/density then we can just make matter on spot and we do not have to travel anywhere anymore, because we can make gold bricks or platinum sieves in particle accelerators just for fun? (this is same argument as why alcubierre drive is nonsense, having capability to manipulate such energy density makes us not want to travel anymore)
Making matter from energy would still be inefficient though. c^2 is a really big number.