having worked specifically on website responsiveness improvement for a medium-size publisher, I can certainly attest that ads heavily influence performance, even if added asynchronously after the first paint, or after the site becomes interactive; we tried every trick in the book, and then some.
the main reason is that ads are coded by people that have no interest in performance. I've seen huge JS libraries loaded to display a static image. I've seen multiple versions of the same library loaded in the same ad. I've seen things that wouldn't fly anywhere else.
Why? Because the ad agencies are under a lot of time pressure to make things quickly, and there is NO penalty if the quality is terrible. So they take what worked last campaign, add new tags to satisfy the new customer, and ship it out. It displays? Perfect. It's huge and slow as hell? Who cares, it's not their website that gets slow as molasses.