Not much to add. This simply isn't the case. You usually get the patent so that you can build it without wasting your time. Alternatively, you get the patent to stop a competitor.