Think of it this way. Ad revenue is based on CPM, or cost per thousand ad views [0], which is itself based on the expected return of an ad view. The expected return obviously depends on how many people make purchases for the products advertised to them. So, using your same logic, you could conclude that refraining from buying a product that you see an advertisement for is causing harm to sites that are ad-supported.
[0] Yes, most web advertising is not CPM. I'm simplifying for the sake of the analogy, and the same analogy holds for other ad models.