This is what makes it so doable since you don't need any privilege escalation.

The reason why this is a big deal for a lot of people is your ssh keys will give you access to your git repos and other servers unless you have them password protected or use gpg/sk ssh keys which I think a lot of people don't do.

And of course if you can see the known hosts file/bash_history you'll likely have access to more servers to propagate to.

Also things like your browser cache is stored there.