There's a big downside, the performance penalty. For a GUI that renders moderately complex objects, the cost of not caching becomes overwhelming, the equivalent of losing 20 years in GL architecture advances. Pushing the VBO to the GPU each frame is the same as losing indexing. My own application doesn't render at 60 fps in immediate mode.
I find that people who believe that IMGUI is somehow faster than RMGUI are game developers who have been taken in by marketing, because basic knowledge of GPU programming (i.e. what is indexed rendering) is enough to see that this couldn't possibly be true. Their UIs are usually simple enough that the performance penalty is not important. And many RMGUIs have heavy styling, visually incomparable to their IMGUI counterparts, which makes the average RMGUI far heavier.