On re-examination of what you wrote I think I misinterpreted this sentence:
"everyone who have a physical key I keep within a reasonable distance from me that is incredibly hard to copy and has my password"
I took (hard to copy and has my password) to be properties you were giving the physical key, but in fact I see the correct interpretation was that "and has my password" is an adjunct to the properties of this hypothetical attacker who now needs to steal the key.
