It does seem like this technique could work with some refinement. Perhaps just making the mobiles about 33% smaller would help.
The fact that multiple camera angles sync a virtual object is really impressive. Keep at it there is something valuable that’s almost captured.