No matter how good security gets, attackers will always adapt. Everyone on earth is now using YubiKeys? Now you need a process in place for when people get their keys lost or stolen. Or when your computer doesn't have a USB port. And whatever "I forget my password"-esque process that is will probably be much easier to attack/manipulate/social engineer than the keys themselves would be.