Ad hominem is a fallacy if you are arguing hypotheticals and philosophy in a Greek salon.
In understanding how the world around us works, credibility matters quite a bit, and "I'm not interested in pretending True Pundit says true things" is a pretty reasonable shortcut.
Rather than just thought-terminate with "logical fallacy," the burden is now on the one bringing the evidence to bring it via a channel other than True Pundit.
