The capacity nosedive from having kids goes away as they grow up and become independent. And the coping skills you learned to deal with having less energy become almost super powers once your energy comes back. Likewise, the skill of having dealt with children through all ages and stages makes it easier to relate to difficult co-workers. I'm not saying work is easier once your kids are older... but your capacity to manage the work and deal with challenges definitely bounces back higher than it started before having kids.

So I'd say follow your dreams early, or follow them late. Just don't try them at the same time as raising kids.