A penalty for filing a bad suit can be (and notionally already is!) part of the price of filing suit, just as overdraft penalties are part of the price of a bank account. A contract between the plaintiff and the defendant is not necessary.

> Loser pays for frivolous lawsuits only (but not all failed lawsuits) is also a thing, and is common in US jurisdictions, but requires a separate court determination

Requiring a separate court determination is a coincidence of the American system, not a logical necessity.