The problem is that there's a sort of rock-paper-scissors game of (conservative)-(greedy)-(all in). That is, a player who build defenses early on will crush a worker rush but they'll be far behind a player who ignores defenses and focuses on growing their economy. The worker rush is designed to punish this greedy build order.
There's nothing that can be done, really. If you decide to send a worker to scout super early on then your economy suffers so much that you'll be behind no matter what you do.
Edit: to those of you drawing a distinction between an immediate worker rush and something like an early pool or a cannon rush, that's splitting hairs. These are all classified as all-in strategies that almost automatically lose if they encounter a conservative build order.