the sim card has one important difference. It lives in a device that provides it with 24/7 battery and radio access.
That is really worrisome when you think about. A tiny computer running applications you have no idea/access. powered 24/7. Always with you. With access to battery, network, mic, etc. And the other side of the network that could monitor it's traffic for malicious actions is owned by the very people that could abuse it in the first place.