If I can't privately listen right now, I don't, and if they get annoyed, I'll tell them why.
If something would require me to replay it multiple times to take notes, depending on the situation I'll tell the sender to type it up.
It isn't a matter of trying to punish, it is negotiating preferences. Send me something I have to interact with, and it may well take me some time to get to it. This is no different than emailing me a PDF.