Most crashes as shipped are just the shader over running maximum time and it being killed, AIUI, rather than actual segmentation faults or attempts to read garbage. Most actual known issues that are potentially security concerns are worked around through preprocessing what the driver gets, and given the level of testing I don't worry about it more than any other OS component security wise. The biggest risk would probably be a new driver release shipping with bug leading to exploitable code being executed.
Also, if I'm not mistaken Chrome shipped first: and they'd done masses of fuzzing of both their code and the drivers. Nobody really rushed into it, everyone keeping it off by default for an unusually long time to catch issues before shipping.