Speaking as a drummer and sometimes producer, the only people likely to have strong opinions about this are drummers. All of my drummer friends agree: audience mixes make some albums unlistenable on headphones. Nobody else will notice either way, so mix drummer perspective on the off chance that the drummers listening will appreciate it.

Live albums get a pass, of course. And yes, lefty drummers exist. If a lefty drummer wants their kit panned the way they hear it, that is of course fine.

yes, you are doing it wrong. drummer is not drumming from the audience perspective

I'm assuming OP is themselves a drummer.