Same with any other loan product the bank doesn't care if it's a scam, only cares about collecting that glorious interest payment or fees associated. Plenty of people buy bad used cars with loans that span over years and that used car may fail months after purchase. Guess who doesn't care... the loan provider/bank.
Still the point is Affirm was the wrong target for their chargeback and the OP should have contacted Affirm to potentially resolve the merchant not delivering the goods or a refund of the loan.
Merchant > Affirm > OP Bank