I've also been self-hosting email for years, and the only deliverability problem I've ever had has been with AT&T. If I try to send something to an AT&T customer, I get an automated "your message has been eaten" notice, and following its directions accomplishes precisely nothing. At this point, I can only guess they're hellbanning the IP block in which my VPS resides, because it does not show up on any public DNSBLs.

Google? No problem. Comcast? No problem. Charter? No problem. AT&T? Problem.