Ultimately, everything is "permanent, unchangeable secret", including private key and biometric data. Where the data is stored and how is it accessed makes all the difference.
I could not find the original definition of "something you have", but modern standards like PCI actually give OTP auth as an example of "something you have" (p. 4 of [1])
(I am not looking at the degenerate case of running TOTP app on the same device / same security domain -- it does not describe most cases, and there are some fairly straightforward technical measures to defeat this)
[1] https://www.pcisecuritystandards.org/pdfs/Multi-Factor-Authe...