It makes sense to me that zip codes don't matter (or might be a weak signal), since some countries might not have postal codes, or might have a different postal code format. But I agree with you that it doesn't make sense to not check the CVV and expiration date; both are printed directly in the card, and should match exactly (unlike the card owner name, which is also printed in the card, but the user might type it differently, for instance typing in full their middle name when it's abbreviated in the card).
Haven't dealt with credit cards, but people often have the zip code wrong on their address. The mail gets delivered if the rest of the information is correct.
I assume that makes it hard to be strict about zip code.