Most of the people running servers are doing it as a desire to be the part of the ecosystem who runs servers. They aren't interested in making money off the ecosystem as much as becoming a part of the ecosystem, and as server ops have been since the first days of IRC and the earliest dialup BBSes decades ago, there's more than a little power trip involved. If they wanted money they'd spend time on mturk and easily make more, unless they somehow go pro and make it into the really big leagues. Much like pro sports.
I'm guessing the pirate effect wouldn't be an issue because the "pro mod writers" would be about 10x more excited about the topic than "pro server operators" for obvious reasons so the mod store writers wouldn't do anything too dumb... I hope. One rather trivial and easy to implement solution is MC currently has a centralized auth system for MC users that existing servers talk to (I have a server in my basement, and I am made well aware from my kids when the MC auth servers are down), and extending that a tiny bit to auth both MC and mods isn't much work other than a bit more traffic to the auth server, which they can probably afford by skimming off mod sale prices. So you can tell the server all day that you have a copy of mod name jiggy-2011, but when the server turns around and asks the existing centralized MC server if username jiggy has actually paid for mod name jiggy-2011, then ...