If the first case delivering the smaller amount and refunding the overpayment would be what the customer would want. In the second case delivering nothing and refunding the entire payment would be what the customer would want.
I could see purchases of financial instruments fall into either group. It would depend on what alternate investments are available to the person. Say I've got $10k to invest and my first choice in investment #1. I try to purchase $10k of #1 and it turns out there is a limit of $5k. My second choice is investment #2 which has no limits.
In that case I'd want $5k in #1 and the other $5k in #2 and so the behavior I'd want from the seller of #1 when I try to buy $10k is to sell me $5k and refund my $5k overpayment.
But suppose #2 has a minimum investment of $6k, and I have no good #3. In this case what I might want is all $10k in #2. When my attempt to purchase $10k of #1 fails I'd want them to sell me nothing and refund all $10k so I can buy #2.