I think it irritates me for the same reason I get irritated by American English speakers calling soccer “football”, or by people using a dieresis when writing “coördinate”.
Basically, it’s rare enough that it doesn’t sound natural and therefore comes off to me as an affectation, and makes the person sound weirdly smug about being “technically correct”.
I really should get over it, but like I said, not really rational.