The mere possibility of having to pay would stifle most such suits, and in many (most?) fields the problem is not too few suits, but not enough suits. Using the legal system to create prospective deterrent effects, as ours does, is not compatible with mechanisms that discourage suits, because ultimately that makes rule breaking profitable for rule breakers.

I also strongly disagree that we need special rules for patent litigation. There is a rhyme and reason to the rules for cost shifting, one that doesn't depend on the subject matter of the litigation. What you seem to be implying is that patent litigants are less likely than other kind of litigants to being meritorious suits, and indeed that a patent suit is more likely than not non-meritorious. If that is the case, then the problem isn't who pays for a losing suit, but the law that creates enough wiggle room that these loser cases are being brought.

The way our litigation system fights meritless litigation is to make it relatively cheap to dispose of meritless suits. As an outsider to the practice, it seems to me that one of the big problems with patent suits in practice is that it's a relatively complex, fact-specific inquiry to determine patent validity. The usual way to address this is to add additional, easy to prove thresholds that let a case be dismissed without consideration of involved questions.