That only works for diffuse lighting. There is more than diffuse lighting. I'm not "narrow minded", I actually work on renderers for a living so I know the actual data structures in use. Things like specular reflections and refractions are entirely dependent on where the viewer is; and calculating lighting information for an entire scene is way less efficient than calculating it for a viewer (see: how deferred renderers work).
> Unfortunately not a lot of research has been done because rendering 3D polygons fit the constraints we had historically and is really robust, everything is optimized towards it.
Huh? There's been decades of research into ray tracing and voxels. Believe me, people have put a lot of thought into how to optimize these things.