It's a design decision and a good one, at that. Forcing 1 million developers to handle spaces properly while they spend 1 billion hours developing their software is preferable to 1 billion regular users having to remember to not use space, a perfectly good separator, in their 1 billion billions hours of usage of said software.

Speculation, but I suspect that Microsoft explicitly put both user data and programs in paths with long filenames and spaces to help force software to support those things. That always seemed like a good thing to me, it's ridiculous that we still have software that breaks on paths with spaces.

I haven't looked in a while, but last I checked you could also still get to it via C:\PROGRA~1? (This was years ago, that might have been 'fixed'?)