The Ledger is designed to plug into an untrusted PC, that's the whole point. It's running secure hardware and never reveals the private key. It also has a display that tells you how much you're sending and to what address, so you're protected even if you have spyware that attempts to spoof those parameters.
According to Yubikey, "All YubiKey NEO devices manufactured as of February 10, 2015 supported the current FIDO U2F specification for NFC."
https://www.yubico.com/products/yubikey-hardware/yubikey-neo...
Maybe you have an older device? Or, if you have an iPhone, it's Apple that's the problem, since it restricts NFC to Apple's own payment system. With Android, NFC is available to any app.