First of all, when assessing security of a solution you have to define the perimeter of attack. You could imagine a privilege escalation that gives you access to the loopback interface but not enough to install a key-logger.

In other words, you are making an assumption that could be wrong!

Second, I could agree that perfect safety against every attack is impossible unless you assume your machine is switched off. But reasonable safety measure is achievable at low cost. Diffie and Hellman won the Turing price yesterday and their work is almost 40 years old...