I'm not sure anyone would ever claim that patents are morally justified. Their justification is cultural, in a sense. They are there that invention is worth it. Invention and innovation, in turn, bring about advances for the entire culture that has surrendered these specific rights, because it sees more value in progress than in the rights that are given up for it. I don't know whether a culture without patents would still advance and have as many inventors, but on the surface, this sounds like a reasonable trade-off.
I would restrict patent law to force a patent holder to license their patents to whomever for a reasonable price. We already have that for standards relevant patents, but i think it should apply to all patents.