Your iPhone is completely covered with your fingerprints. Also, if your phone gets hacked/compromised the attacker could steal the secret used to generate your TOTPs. This is not possible with the Yubikey, it is absolutely impossible to extract the private key.
I have a Yubikey on my keychain (it can easily withstand this), and it takes very little effort to plug it into the USB port when I require it. Less than it would be to take my phone out.
As a side-note, some time ago the Yubikey had a vulnerability with its GPG module so they shipped out new ones for free. I now have the old key (with no GPG keys loaded on it) permanently plugged into my USB hub at my desktop. It is amazingly convenient.