Oh dear. Video acceleration means GPU is doing what CPU would be else doing. Which means user threads get less indirect access to GPU. Which is a potential security risk, because GPUs and GL are designed to perform, not to secure data.
VA-API is not mentioned in the question at all. It is the most common api for video acceleration, but there are others - direct vdpau, or nvdec.