I have SSHed into a remote server from my phone out of necessity many, many times. When you get that e-mail on debian-security-announce that some package on one of your servers has just received a security update to fix some serious vulnerability, then ideally you install that update and reload the service right away, instead of putting it off for later.
Is this for a personal server or for a server that is being used by a paying client? If the latter, doesn't it make sense to have an oncall rotation with someone else on your team?