This is a very good point. Essentially what you are describing here is a certificate authority.
The Yubikey, in this scenario, just acting as an 'offline CA'
It's a very good idea, but requires software being built to accept an authentication hierarchy.