For video ideally you want both the shutter speed and aperture fixed since having the depth of field fluctuate in the video isn't ideal either.
I once thought similarly to you are thinking now. I quickly gave that up for the specific type of shooting discussed. You quickly realize that the software in post for stitching is really good at what it does.