While the displays are significant areas of power draw, for this product I don't think they're the low-hanging fruit. This is a device that's performing a whole bunch of computation in real time: video, lidar, infrared, eye tracking, and then it has to drive two 4K+ displays, one for each eye, with 3D-accelerated content being rendered at all times.
Go to an AR web page on your phone and start playing around with 3D objects in your space and you'll notice your phone getting significantly warmer and drawing more power. The Vision Pro is doing this literally all the time.
Also, the camera and sensor work that is tracking your eyes has to happen whether or not there is an outward-facing display.
Apple makes a watch with a display that is always on, and their phones have high resolution OLEDs that can stay on for over 13 hours (iPhone 14 Pro Max 150 nits brightness doing continuous 5G web browsing).
