The downside is, the types of skills and mindsets you have to cultivate to make a lot of money might actually be deleterious to being able to succeed or thrive in other fields.
I think of a lot of people who came up in the management consulting meat grinder and left. Even though they almost all hated the work style and cultures of the firms they left, they often cannot help but replicate the dysfunctional cultural tendencies and management styles that contributed to their burnout in every other place they go. There's a lot you have to unlearn before you can really pivot, but when you've been trained early on that certain inputs are your ticket to praise and promotion it's hard to think any other way.