Generalized scene understanding is still a hard problem in both academia and industry. The big companies are working on such things with access to tons of data.

Assuming you do know what’s in the scene semantically most ar applications that aren’t games do intend to use that information. I’d say the primary hurdle there is still general scene understanding.

It’s easy to identify a sudoku puzzle but not so easy to identify and classify all 3D objects with any sense of precision yet. Seems like a large scale data play to me..

You’re idea is to build that context layer? It seems the apps you’re talking about will become reality when we have useful AR glasses. But they will be manually built for each usecase.