I'm not up to date on VR tech - is the video memory on board the VR headset? If the memory is onboard, you could do some simple rotations and translations on the current frame while the render pipeline caught up.