Doesn't it depend on who's in the audience? You make a joke at someone's face, you're the baddie. You make a joke in a show I specifically went to go see you for, or I clicked on your recording to watch you, it's on me.

Generically and universally decrying punching down is extreme and would require spying/policing everyone to implement.