I'm similarly racking my brain, and I came to the same finding.
755 permissions on the home directory lets others see what you have, which isn't great.
The good and bad news is, permissions on the files matter too.
SSH (private) keys for example categorically won't work outside of 600 permissions, meaning nobody else can read your private key - without escalating privileges
Now, if you go defining auth secrets in your shell profile (which is world-readable by default), probably something to reconsider.
Restricting umask is a good protection for this, for what it's worth. You can make it so that newly created files/directories are not accessible to the world
