David MacKay in "Sustainable Energy: Without the Hot Air" did a calculation circa 2010. To fulfill the world's energy needs back then, a 10 km^2 area in the Sahara desert would be sufficient. Even if you scaled that to 100 km^2, it's absolutely tiny on a global scale, and panels have only become more efficient since then.
The challenge of course is storage and distribution, but yeah, in terms of land area, it's not much.