Just as some examples.... we've moved some pages to vanilla jQuery to Angular and have gained orders of magnitude more stability (read: basically no state-inconsistency bugs in newer pages).
If you're faced with vanilla javascript in the DOM and have any amount of complexity to it, then you should be using angular/react, if not for your sanity than for the sanity of the one who will need to maintain your code (read: you in 3 months).
If you're knowledgable enough to know that your use-case doesn't work with the newer tools, then more power to you. But the productivity gains from moving away from your jQuery spaghetti code to one of the more managed frameworks is well worth the rewrite time.
I don't know much about going from Angular to React, or Backbone to Angular though.... the choice of framework is likely dependent on what kind of user interactions dominate your app and subjective comfort.
But if you're careful, you can actually transition apps slowly over to Angular and I assure you, you could get rid of so much of your custom-written (probably not well tested) DOM management code it's not even funny.