It seems to me you are ascribing properties to "something you have" that aren't warranted. The "something you have" needs to prove you were party to the initial exchange, not necessarily that you were the only one present -- that's why we use two factors, and not only TOTP.