I don't know if XUL is the right choice, but I think this general direction might be a good idea.
The idea of an independant open source project focusing itself on producing a commercial quality UI seems a little ludicrous to me. This type of UI requires continued multi-disciplinary effort and process over a timeframe of years. And this is the exact weak spot of projects funded entirely/mostly on a volunteer/contribution basis.
This idea of a highly customizable toolkit (a la Emacs for graphics), in contrast, goes to the strengths of the open source model. Open source projects tend to be developer-heavy, and developers tend to like to build frameworks/abstractions... so a natural fit. Frameworks tend to be easier to use when the underlying code is available for various kinds of inspection, so the open source model fits there too. I'm not saying it'd necessarily be a cakewalk, but it seems like a closer fit to that OSS is really good at.
(Of course, frameworks are more complex and expensive to develop than single applications, so this idea has that going against it.)