The problem is that your home directory is accessible to a ton of apps on your computer, and you have no idea what each of them is doing with that access. You also have no idea if any of them can be / are being exploited. The most recent case being Zoom – if that server they had running on localhost and responding to anyone on the laptop had file system access APIs (which is reasonable if Zoom had offered file sharing on calls) an attacker would have been able to read all your credentials.