Except... the card issuers will almost always side with a customer over a merchant in a dispute. They still have to worry about that credit risk.
"Give me a refund or I'll initiate a chargeback!" is a threat with a lot of weight behind it. Mutiple chargebacks and your rate goes higher or you're just dropped altogether.