Government work is hard for the reasons you mentioned, and also because generally, contracting pays well enough to price you out of other markets.

My exit strategy was to find a government vendor (e.g., that makes products for government / enterprise consumption) vs. a government contractor. The services side of the house is still pretty much a suit-and-tie affair, though we pride ourselves on being much more meritocratic than our customers, but on the engineering side, pretty much anything goes.

The one thing I will say about gov contracting is that as a result, I interfaced with much more than would have been in my normal comfort zone, and I came away from it MUCH more knowledgeable than I would have had I not done it at all, and the work was very rewarding. My advice is to hold out for the good parts as long as you can while quietly looking for a way out that seems fulfilling.