I think you overstate the power of Patent Office. I don't think they can just say "Turns out we shouldn't have granted you that patent in the first place." Without providing evidence such as prior art. I don't think the courts would tolerate they are making too much money" argument from the patent office.