Being comfortable is far better for confidence than "dressing well" in my experience. I would never wear shoes in my home and I work in pyjamas mostly.
However if shoes or clothes make you comfortable then by all means, but I doubt this makes a difference for everyone.