You raise a good point about the limits -- although the contactless limits have been raised here due to the pandemic. But I think you're underestimating the physical security of the payment infrastructure. Card terminals (here at least) are extremely secure. They are highly tamper resistant; they cannot access the network if they've been tampered with.

These NFC terminals on the other hand, I have my doubts about many of them. Especially the ones that are basically glorified phones.

Stealing a PIN and a card would be a targeted attack (or an extremely lucky break). When that happens, depending on the bank, taking out money might even be dumbest course of action. Many banks have online services that can be accessed using a card reader (and the card, and the pin). Limits there tend to be much higher, and there won't be an ATM camera filming you.