Of course you take it off when you get home... so? Yes, it is hard not to touch your face, which is why having a mask on it helps you remember not to... just try it. You have to consciously reach under your mask to achieve skin to skin contact. You can easily scratch your face through your mask without touching your face. Touching your mask is not the same thing as touching your face.
I’m rather confused about this push back. Of course wearing a mask doesn’t guarantee you can’t get infected, but it’s better than not wearing one. There’s a bunch of evidence of this, in studies, and in China right now. What is there to worry about? If you’re really worried, then don’t go out.