It's not a password, it is a secondary, single-use recovery second factor.
Carrying that around in a wallet doesn't make you any more vulnerable to physical attackers than carrying your Yubikey on a keyring, and it's much more secure against remote attacks than SMS-2FA (where you can fall victim to SIM swapping, number porting attacks etc).