True, but imagine this use case:
A messenger file with hardcoded settings and a hardcoded PGP key, stored on a USB stick.
You send a message.
Then you physically destroy the USB stick.
The client, the key, and the configuration are gone.
At some point the joke starts looking suspiciously like a dead-drop communication protocol.
How do you like that, FBI?