The problem may be too much investment in drivers. Modern graphics card drivers are little operating systems unto themselves; they even have built-in compilers. They are incredibly complex, yet written in unsafe languages for maximum performance, with tons of ugly hacks for compatibility.

What we really need is a simplification of graphics APIs so that graphics drivers don't need to be so complex. That may be impossible without convergence in hardware though.