VR is cool, but it's seems like such a more useful concept to me to have actual reality with enhanced information.
Imagine wearing glasses and looking at a plate of food then having it estimate + track calories and macros, or paint GPS direction arrows on surfaces realtime, or put people's names you've met before above their head so you can avoid awkwardly admitting you've forgotten it.
Is it technical limitations, or cost?
---
Edit: Many people replied with really informative answers to this already. I genuinely appreciate your time and insight, thank you :)
There used to be a good blog post from Michael Abrash when he was at Valve that also talked about two main issues. Latency, and drawing black effectively.
Latency is critical since low latency is a requirement for things looking real (since humans have fast visual systems), but that's ultimately a hardware problem that should get solved in time.
Drawing black is harder because AR uses ambient light and putting a black line on the screen in front of your face doesn't work for focus.
Unfortunately it looks like Valve killed their blog, but the way back machine has it: https://web.archive.org/web/20200503055607/http://blogs.valv...
My bet is that Apple will pull it off Apple watch style with front facing Lidar: https://www.youtube.com/watch?v=r5J_6oMMG7Y
Probably at first they will be mostly for notifications and interacting with apps in a window in your visual field, getting most of the power from the phone. Things like looking at food for calories and names, etc. will come later when a front facing camera is acceptable and there's existing UI in place.
I think this is probably the next platform after mobile devices, looking at little glass displays is a lot worse than having a UI in your visual field (if it can be done well).
[Edit]: A more recent blog post from Abrash on this topic https://www.oculus.com/blog/inventing-the-future/
Of course you need a lot of space to route incoming light through that focal plane and then to your eyes.
Because demand is too low, and prior attempts at hyping it up with marketing ended like google glass?
I do work in an engineering consultancy that did a dozen of VR/AR toys over the last 5 years.
Some of quite big brands around use our tech, and engineering, though NDAs, NDAs, NDAs...
There is no magic trick behind any of product on the market. Physics, and optics of AR/VR glasses is very simple, high school level simple. Just too many companies want to make add "smoke, and mirrors" into the optical scheme...
Making AR/VR goggles power efficient, and lightweight enough for daily use is possible even with current day tech. It's not much of a secret now that there is an IP blocker on a critical technology owned by Beijing University that shuts everything down.
Microsoft, Facebook, Apple experimenting with lasers now is all about them trying to work around that blocker.
It's coming but everyone is waiting for the tech to be ready.
Everyone remembers the "glasshole" problem with Google Glass.
What keeps costs up are the technical limitations. Microsoft and Magicleap invested both more than a billion each to solve the tech challenges, but the truth is that the displays are not good enough. Framerate is too low (compare with VR what would be necessary), FoV still rather limiting, colors and blacks are still to faint for many environment lighting conditions, room scanning is still too inaccurate, battery life too short.
This is not to say there hasn't been great progress. Hololens 2 solves the comfort problem and is a nice step with resolution and FoV. They just messed up with image quality (color banding/rainbows are a big issue).
Just think of how it feels to look at your phone halfway through a long hike. I always feel it lessens the magic, and introduces low grade anxiety.
The hardware is significantly harder to get right. VR devices work well specifically because they don't care about your environment. AR devices, to be any good, need to have a semantic understanding of your environment. That's very hard to do, especially on the power and compute budgets that mobile devices allow. And an AR device that isn't mobile is a stupid AR device.
The software is significantly harder to get right. It's a lot easier to model a static scene and make some physics-based interactions in it than it is to try to figure out how to make overlays that react to a real environment.
But I think, much more importantly, the people producing most software in the immersive software space just don't care about your immediate environment. They care a lot more about giving you a canned experience. It's hard to find funding for anything that isn't some sort of media consumption. This is true across both VR and AR. In VR it's ok, because there is no environmental context to exploit anyway. But on AR devices, you just end up with a bad VR app: all the hardware limitations of a mobile AR device with none of the differentiating features. So because you don't get software that cares about your environment, you don't get good user experiences on the hardware.
Open up your iPad or HoloLens or Magic Leap app stores and take a survey of apps that are there. How many of them have any understanding of your environment? There are a lot that don't even take into account the "room mesh", the solid surfaces that the device can see--say nothing about what those solid surfaces represent! I'd estimate it's upwards of 50% on AR headsets and maybe 30% on iPad that do absolutely nothing with any surfaces beyond asking you to find a flat space on the floor. That's just crappy VR. As for the ones that attempt to understand what is in your room? Vanishingly few.
You can make pretty good consulting career out of making what is largely just a PowerPoint presentation in 3D: a collection of canned elements that the user can click buttons to get scene transitions and animations, all with a directed narrative that is trying to tell you something. Advertisers want it. Media companies want it. A lot of big-industry companies completely unrelated to media want it just to show off at conferences to "prove" they are "forward looking".
And you'll get a lot of those clients asking--even demanding--you make that as an AR app, especially on iPads. But it sucks. It's just not anything about what's good in immersive experiences. It fits a little better in VR. It still sucks in VR. But it comes around from backwards priorities. These companies start from wanting VR/AR and work backwards to a use-case. And often they lack any sort of experience or even actual interest in immersive design. What they want is to just do a marketing piece. There are very few companies that start with a use case and then find out whether or not VR or AR is the right solution.
But that's where the bread-and-butter money is. And it sucks the air out of the room. It leaves the real, good, immersive experience development to people who are independently wealthy enough to do it on their own, or to hobbyists hacking it together in their spare time.
Do you have any people or communities or reading materials you'd recommend to get more familiar with AR in industry?
VR is better because you can fabricate entire worlds and spaces for any task. Instantly useful. And if you really needed to combine the real world with generated content, you could theoretically do it by just overlaying content onto a video feed in VR.
I played with the Hololens 2 last year and I certainly can see the benefits. Especially when communicating with someone else.
However for $6000 that device is not going to happen for me ;) AR needs an "Oculus Quest" approach to be useful to consumers first.
I do love VR as well but I do think there is a big usecase for AR once it becomes good and affordable at the same time.
He's an engineer that understands the limitations of physics, especially when it comes to optics and light. He is very good a parsing marketing hype vs reality. (He called BS on Magic Leap years before it launched.)
His posts are exceptionally well researched and explained, even if you don't have a background in physics or optics.
He recently did an analysis of the Apple Glass leaks. I expect he'll post his thoughts on this new technology from FB soon.
I've been reading Karl's takedowns for a few years now and, while there is never anything technically wrong about what he says, it's also just not as important as he thinks it is.
Yes, waveguide optics aren't the best possible visual experience once could have. But does that matter all that much? I think Karl's terrible diagrams point out why he makes this mistake. He doesn't understand that design is the much more important consideration here.
The technical limitations of the display technologies that we have today are not impediments to product development. Good software can be designed to work around these issues. Can't render black? Don't design around dark themes. Have a narrow field of view? Don't require people to try to keep mental track of things around them by vision alone.
Karl looks at the HoloLens, sees the waveguides, and misses all the amazing operating system features. Speech recognition, spatialized audio, a fully spatialized desktop metaphor. These things are important and they go a long way towards the usability of the system.
And that was why the Magic Leap failed. Not because the displays were crap, but because the entire system was crap. It was basically "just a" stock Android system with super flaky WiFi and no systems view on delivering a unified product. The entire product was fundamentally mismanaged. The hardware was slightly better than the first HoloLens, but you were far more limited in making good software for the Magic Leap than you were for the HoloLens.
Edit
i suppose parent was downvoted bc some thought he was questioning the "hard" laws of physics. but perhaps he could correct this by traveling back in time and changing his wording ;-)
Fundamental skepticism is nice in a thought experiment, but what value are you trying to add with such a vague statement?
1. Weight and overall comfort. Pressure on the forehead, sweating if you're playing physical games and the room is warm can make the lens foggy; even the rubbery-plasticky smell of it. It's just not fun to use physically.
2. Resolution/graphics. I just wish it could have higher resolution and I won't see the pixels. Would make everything much more immersive.
3. Usefulness/content. It's cool but it gets old pretty fast. I just found myself barely playing after a couple of weeks of the initial excitement.
Motion sickness wasn't a problem for most games, which was a surprise since I get game sickness from FP games on a normal screen and can't play them more than a couple of minutes.
So I guess 1 and 2 can be solved by this technology.
I know this may not seem like "intended use case", but the developer experience can use some innovation for a change. Also one way to bring these technologies close to developers.
Roughly, per-eye resolution is in the same ballpark as HD displays, but stretched over a 90+ degree field of view. Fonts need to be very large to be legible. You can create a theater sized virtual monitor, but it's just taxing to use. Aliasing artifacts make it worse.
At least for text-focused tasks, I'd take virtually any display built in the past 40 years over a modern VR headset.
Spoken like someone who hasn't programmed in VR yet.
I don't think you'll be programming any operating systems in VR anytime soon, but there is still a lot of programming, specifically object scripting, that could be done in VR. A number of people--including myself--have built demos that prove out the concept.
One of the reasons is that text legibility is not strictly about display resolution. Motion within the view improves legibility significantly. Yes, the fonts render to very large pixels. But the specific pixels they render to are constantly changing. Your brain fuses those images over time. I'm not able to find the paper right now, but the US Navy did a study that proved pilot visual acuity improved when they were in a dynamic scenario. The study performed a visual acuity test where pilots had to identify letters in view from within a flight simulator. One group had full use of the simulator in motion, one was told the simulator motion systems were broken, but they still sat in it to perform the same test rendered on the same screen.
And as you said, larger fonts are easier to read. There is a lot of spatial resolution in VR that is not used very often. You're used to thinking about organizing your code on a 2D display, but you have an entire 3D environment around you. That environment could be a zoomable interface where code editors are linked to live objects. Use individual editors for individual code units. Organize them in a tree structure linked to the object. Trees organizers are a lot easier to navigate in 3D than on a 2D screen, especially if you eliminate window scrolling.
Window scrolling was created to account for the limited spatial resolution of 2D displays. But in the process, you lose spatial memory of where things are located. Things like windows and tabs and desktop workspaces were invented to try to wrangle that problem more, but they are not as good as a real, spatial filing system.
Think about it. You probably know exactly where your favorite book is on your bookshelf. You could probably walk over to it and pick it off the shelf without even opening your eyes. But there is very little chance you can pick any particular file you want in a 2D GUI system, specifically because of the absence of spatial relationships.
So a combination of "text legibility is not as bad as you think it is" and "code could be a lot more organized than it is on 2D displays" means that programming in VR is a lot better than you're making it out to be.
https://research.fb.com/wp-content/uploads/2020/06/Holograph...
“Finally, holographic lenses can be constructed so that the lens pro- files can be independently controlled for each of the three color primaries, giving more degrees of freedom than refractive designs. When used with the requisite laser illumination, the displays also have a very large color gamut.”
They have full colour working, and apparently well, in the benchtop prototype.
There’s discussion towards the end as to options for implementing full colour in the HMD prototype.
Thanks!
Digilens is a company that's been around for 15 years doing all kinds of eye wear stuff.
If you are building an AR system there is always an awkward balance between "letting the environment shine through" and "having projected items be bright enough to be visible". If you put something black in front of the holograms at least now you have just one problem instead of two problems.
The intensity of light is quite high outside, could an LCD not darken it enough to have sufficient contrast if you were OK with it only working outside?
1. https://www.researchgate.net/publication/266659406_Pinlight_...
I suppose you could incorporate both.
