You can host your email yourself (and there are a couple of email-in-a-box bundles out there), but you will find that reliable deliverability takes surprisingly much effort. (Distressingly much effort.) You can count on winding up on various blacklists, even if you can’t imagine why it would be so; so you’ll need to monitor them with things like MxToolbox, and then manually sort them out—and/or maintain a pool of IP addresses to rotate through (some providers make it very hard to resolve). The big providers are often not fond of low-traffic senders, so you may have difficulty there too. And every so often there will be new hoops for you to jump through—SPF, DKIM, DMARC and so forth; receivers will change their rules and all of a sudden your email won’t get through any more.

Of course, it depends on how you use it. If you send out a newsletter or other large quantities of emails you’re in for much pain, while if you tend to only send individual messages to a comparatively small group of people (and are thus a known sender) you’re likely to have an easier time of it—though you may still have some trouble occasionally.

(I’m a FastMail employee, but I don’t work on the backend side of things at all; I speak from casual experience mostly before FastMail. But I think that all of us that know anything about how hard it is to run your own mail server are ideologically sad about it; if FastMail wasn’t around, I suspect a substantial fraction of us would run our own mail server.)