Bots already exist, so the foundation for automated play testing is in place. Take the basic AI and add some functionality to track the effectiveness of various skills or loadout across plays. Using A/B/n testing to choose the most effective character strategy would probably highlight overpowered loadouts within a few thousand game-test-hours.

They could probably take analytics from real players and do what's outlined above and get a reasonable idea of the impact a change will have.

"Early Access" seems to be the more popular route now. People are happy to pay to test a game and feel like they participated in its development. Check out Star Citizen if you want to be amazed and depressed by this truth at the same time.

Sometimes it works well. Sometimes it doesn't. Seems largely dependent on how engaged the devs are with their community.