Yes, as it turns out using the internet requires reliance on third parties. Is the IP address of your mailserver as easy to compromise as your domain name? I don't know, it depends on your ISP, but it almost certainly requires a targeted attack. Using email to "secure" access to accounts is already "a really bad idea", so what can one do?