> …it's a little bit refreshing to know that Apple recognized pass-through HMD AR as too hard and decided to invest in compensatory technology…
I understand the framing as "compensatory technology", but is it possible that what Apple's doing is the simpler and better way to solve the problem? Pass-through AR strikes me as an old-school analog approach, like optical printing for special effects. But a 100% digital vision pipeline seems like it could unlock interesting capabilities like "night vision", new ways of highlighting interesting objects, etc.
The other problem, of course, is that nothing can be truly solidly-colored. Everything has some opacity - which, combined with the FOV issue, is why HoloLens was never marketed as having anything to do with VR.
I think they're different and not one better than the other. If I wanted to drive with an HMD on, or otherwise be in a situation where it could be deadly to have my sight turned off for even a second or to even have some lag or stutter or other glitch in my eyesight, I'd much rather have a pass-through AR HMD. One's sense of sight seems much more reliable with it by its very nature. You simply don't have those modes of failure with transparent plastic, no matter what's going on in the hardware/software.
HOWEVER if someone can get the input -> photon production pipeline to be less than ~10ms, then that does solve a lot of the rendering issues, however it doesn't solve all of the other long term problems that come with that amount of hardware - including weight and complexity.
That said, there's a lot of known-unknowns that need to be solved, for example I don't have a solution for micropiezo resonance issues that I'm sure will crop up.
Currently, NVGs that are fielded by soldiers are already displaying images in a way that's not pass-through (using classic image intensification tubes). Something like the Apple vision headset in a lighter and more durable form factor would allow for eg; fusion imagery (fusing visible, thermal, and night vision).
But otherwise I agree, it makes sense for them to focus on what they can do best with current tech as a stopgap. With current see-through HMD tech, AR ends up incredibly disappointing. (See also: Hololens & Magic Leap's limited FOV)
The blocking all light approach also allows you to hide other potential weaknesses of a device. For example, a lower field of view is much more distracting in a pass-through AR device as you still have your full peripheral vision. VR devices will generally black out the light outside of the FOV making it easier to ignore.
1. Field of view is limited to existing optics miniaturization
2. Subtractive shading (rendering black) might not be solvable
3. Variable focus objects in the same scene requires projecting n>2 significantly different wavefronts - not solved how to do this with a single vibrating element
Moore's law works great for semiconductors, but Maxwell doesn't negotiate ;)
And it seems this is such an important aspect of the product that they're willing to reduce the addressable market from a cost perspective.
This, to me, is what makes this product intriguing. And it makes me think that Apple's real goal is something closer to a pair of glasses, and they just know they can't get there without a long series of iterations.
But that display increases the "creepy factor" by orders of magnitude.
apple has decided VR zombies hurt their brand & they won't allow it
you see, we wear a noise cancelling headset to pretend to be working, in order to avoid unwanted socialization.
Early reviewers seem to say that the metal construction of the Vision Pro seems to be contributing a lot to its weight. Most other headsets are all plastic.
In the long run, adding a second screen isn't that expensive, and the cameras that capture the video of your eyes already have to be inside the system to perform eye tracking. If smartphone manufacturers can make folding phones with second screens for under $1000 I think that the outward-facing display is not the lowest hanging fruit for cost reduction.
They’re always evaluating if they can cut a corner.
Apple's solution works for >1 people at the same time and doesn't require any external tracking (though it's already doing the external tracking regardless), at the cost of lower resolution and only being correct in one dimension vs two.
https://research.facebook.com/blog/2021/08/display-systems-r...
> There are several established ways to display 3D images. For this research, we used a microlens-array light field display because it’s thin, simple to construct, and based on existing consumer LCD technology. These displays use a tiny grid of lenses that send light from different LCD pixels out in different directions, with the effect that an observer sees a different image when looking at the display from different directions. The perspective of the images shift naturally so that any number of people in the room can look at the light field display and see the correct perspective for their location.
> As with any early stage research prototype, this hardware still carries significant limitations: First, the viewing angle can’t be too severe, and second, the prototype can only show objects in sharp focus that are within a few centimeters of the physical screen surface. Conversations take place face-to-face, which naturally limits reverse passthrough viewing angles. And the wearer’s face is only a few centimeters from the physical screen surface, so the technology works well for this case — and will work even better if VR headsets continue to shrink in size, using methods such as holographic optics.
The Wikipedia article might explain it better: https://en.wikipedia.org/wiki/Lenticular_lens
It could work in both dimensions but you're sacrificing even more resolution by doing it that way. For example imagine you have a 1000x1000 pixel display (I just made this resolution up) and you stick a 1D lenticular screen on top with a pitch of 10 pixels. You've effectively split the display into 10 separate 100x1000 displays that are each view from a different angle. You could instead use a 2D lenticular screen and split it up into 100 100x100 displays viewable from a different angle in a 10x10 grid at virtually no extra $ cost. However, you're displaying at 1/10th the resolution just to be able to support perspective-correct views from above or below, which are way less common than from the side.
I'm guessing (?) Apple's approach is similar.
¹ https://www.laptopmag.com/news/facebooks-bizarre-reverse-pas...
They also add the display that would work with different angles. So it looks like, maybe Apple implemented Meta's research. The timeline could work.
For completeness, there is also another paper "FrontFace" proposing a similar idea that was published around the same time: https://dl.acm.org/doi/10.1145/3098279.3098548
U do like the LMU after all I'm in Munich but this though is more obvious than magic.
To the best of my knowledge, this is the first work that proposes putting a photorealistic, perspective corrected face on a VR headset.
"FrontFace" (https://dl.acm.org/doi/10.1145/3098279.3098548) is the first work that proposes putting eyes on a display on VR to "lower the communication barrier".
And I am scared of what it means for our society when "eye contact" no longer means a direct connection in person, but an indirect one through 2 cameras and 2 screens.
Obviously this is old hat for Facetime, and all remote collaboration. But in-person too?
And I am scared of what it means for our society when "eye contact" no longer means a direct connection in person, but an indirect one through 2 cameras and 2 screens.
Obviously this is old hat for Facetime, and all remote collaboration. But in-person too?
