After self-hosting my email for twenty-three years I have thrown in the towel (opens in new tab)

The sweet spot for having control over your email while simultaneously minimizing unforseen headaches is to simply own your domain name and point the MX record to whatever hosting provider you want instead of self-hosting a server at home.

Same philosophy for exposing a your personal blog of html files or content like mp4 videos. The sweet spot is to focus on buying a domain name you control. Then let Amazon S3, or Cloudflare, Hezner etc, host your html or mp4 files.

I quit self-hosting email at home over 15 years ago. It's just not something I want to babysit anymore because I have other things to focus on. As long as I control the MX record on my own domain, that's really all that's necessary.

I've been hosting my mail for 20+ years now, with minor issues. I guess I've been lucky.

Reading the comments here makes me incredibly sad. Every answer that tells me to use a provider misses the point. The Internet was created so that there could be many independent nodes, not so that everybody has to rely on one of several blessed providers. I should be able to run my own E-mail.

The real problem is lack of incentives. The big corps do not care about e-mail. It doesn't make money and isn't easily controllable. You can't turn it into a walled garden and lock users in. So, it gets minimal attention, and only defensive measures are developed.

Either we solve the spam problem, or things will get worse. The big tech corps won't solve it for us.

I'm on 12 years of self hosting email and counting. Once every so often, I do end up being blocked, usually by Outlook and once by Yahoo. I'm in their 'sender program' and they still don't actually bother to contact postmaster@, but a few emails is usually enough to unblock the block within 24h.

Agree with a sibling comment that many major providers fail to operate the SPF/DKIM/DMARC tools they insist you do.

Each to their own, but ultimately if we don't hold on to the freedom to operate our own mailservers, it will be taken away through inaction. This means doing some things right: DMARC, DKIM, SPF of course, server maintenance, good password policies and of course IP reputation. The best way I can recommend for IP reputation is to use a dedicated provider or VPS provider that disallows things like VPN endpoints, where it is less likely they'll assign an address with a poor reputation. A good provider might also ask you what you intend to host, and you might be able to discuss IP addresses with them.

For those who might choose to run their own email these days, you don't have to postfix + dovecot:

https://github.com/foxcpp/maddy

https://blitiri.com.ar/p/chasquid/

These options are much easier to set up, will do things like generate DKIM for you, etc.

I talk about this a lot[0]. There are positively awesome tools for email out there.

[EDIT] - Since I'm repeating myself I've collected all the options into a post[1] I can just link to.

[0]: https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

[1]: https://vadosware.io/post/its-never-been-easier-or-harder-to...

It was a huge mistake for email receivers to take on the cost of filtering spam. Of course given the evolution of the internet and email it is easy to see how that mistake happened. Nobody had a crystal ball. But the only solution here is to raise the cost of sending email to the point where spam is no longer profitable.

It seems like one solution is to bcrypt hash (or some similarly expensive algorithm) the email and include the hash in a header. Of course you need to hash per receiver or a spammer can just hash it once and spam away.

The receiving client hashes the email and compares the result with the value in the header and discards emails that don't match.

You'll never get industry buy in though - the FAANG companies don't want to pay that cost for their semi-legitimate email. They prefer to keep that cost externalized.

I believe there have been attempts at something like this, but it clearly never went anywhere.

Sending email out is a royal pain. Trying to deal with a Microsoft ban on my IP even though it’s sparkling clean for several years. DKIM, DMARC, SPF etc all ser up, reverse dns, you name it. Looks like Linode is being blocked as a whole pretty much?

Hate the level of centralization, particularly since there’s still a shit ton of spam still around. Sorry for the rant.

https://docs.microsoft.com/en-us/answers/questions/674558/55...

Maybe there needs to be a self-hosting association/union that self-hosters can join? It could advocate for adherence to open standards, and an equal standing for small servers. It could also be a repository of advice for current best practise in small server administration and configuration. Should it be under the auspices of an existing group such as FreedomBox?

All that "security" just to fight spam. IIRC it was estimated that globally spammers make $300M per year from their spam. It doesn't seem like much. Somebody joked that it would be better if we just paid them that much to do nothing.

>I implemented all the acronyms, secured antispam measures, verified my domain, made sure my server is neither breached nor used to relay actual spam, added new servers with supposedly clean IPs from reputable providers, tried all the silver bullets recommended by Hacker News, used kafkaesque request forms to prove legitimity, contacted the admins of some blacklists.

I cloned a repo, edited two lines in a yaml file, ran docker-compose, logged into a web ui, added my domain, added a couple of dns records (MX, spf, dkim, dmarc) and everything worked (yes, I can deliver emails to gmail and outlook).

I honestly have no idea why so many people say that self hosting emails is hard.

I support the author but let me tell you a counterargument I don’t think he devotes enough to:

Spam is a real issue.

The amount of spam emails which get sent are absurd and likely orders of magnitude more than non-spam. And spammers do a lot to mimic real emails, including just hacking legitimate addresses and adding them to botnets.

Even on gmail, I still get spam sent to my inbox. Fortunately very rarely, but it still happens.

And even if it isn’t bad today, spam has the potential to be much worse in the future with transformer networks and hostile state actors.

And even if it really isn’t that bad and never will be, the big companies and those arguing against self-hosting will claim it is. They don’t want to allow a relative few self-hosted email servers in exchange for much more difficult and less effective spam detection. Forget Gmail and Outlook, why not just use Fastmail or Protonmail?

If you want a legitimate argument for self-hosted emails you need to address the spam. It may be as simple as registering your official email with some organization sponsored by open-source, and all the big companies can trust that one organization. Then the org has to deal with spam registrations but maybe there won’t be much and it will work out. idk much about self-hosting so this org might already exist.

But this article doesn’t mention that org, in fact doesn’t say much at all about spam besides “keep existing spam-prevention because it already works”. But you should at least explain why. Because spam is a legitimate argument for big-co forming an oligarchy that’s not just “so they can make more money”, and it’s the main argument that big-co uses.

I am not a Google employee, but I do work with email anti-spam at scale. There's a lot to critique here but it boils down to three points:

1. Spam filter behavior has changed because spam has increased in volume and sophistication, not because ISPs want to save money, or to eliminate competition. Some techniques that worked well 5 years ago aren't as effective anymore. One of the consequences of this has been a reduction in the value of IP reputation, from a spam signal perspective, particularly for low-volume IPs.

2. IP range reputation does matter. The increase in the value of IP range reputation, as a spam signal, has paralleled the decline in value of low-volume IP reputation. In practice, this means you need to either send enough volume to outweigh the reputation of your IP range (exact quantity varies based on a lot of variables, but as a very rough approximation, 1000 messages a day), or find an IP range with good reputation.

IP range reputation is not easy to assess, sometimes even for email professionals. So you can either gamble with a residential ISP IP, or a VPS IP, or you can find a provider that spends time, effort, and expertise on managing IP range reputation. The practical solution for most senders is the latter. Many of these offer a free tier, and many options are available among providers of all sizes.

3. The filtering behavior reported here is either misunderstood or misrepresented. First, no, no major ISP (Gmail, Yahoo, Microsoft/Outlook, icloud) is going to permanently block an IP range; filters are designed to be dynamic. In severe, ongoing, high-volume spam scenarios, you could see a 2-week block, maybe occasionally 30 days. But never "one strike".

Mail deletion without a bounce also cam happen, particularly at Microsoft, but again it's almost never seen for legitimate mail - that response is reserved for long-term, severe spam scenarios, where anyone reasonable would agree that a block is warranted. And, again, this is dynamic.

So it looks like OP is either exaggerating, or has been trying to send from IP ranges with unusually bad spam problems.

I tried to operate my own email server as well on a VPS, and I have been thinking that the way to solve these problems is to solve the problem of spam itself. Detecting spam puts the costs on the email providers... when the costs should be born by spammers.

Perhaps some sort of digital stamp (digital signatures similar to stamps on physical envelopes) for each email sent paid for with micropayments in a cryptocurrency like nano (note: I don't own any crypto). Small cost per email like 0.01 cents that is trivial for legitimate senders but not for bulk-sending spammers. SMTP servers should put all incoming unsigned emails into spam folders. This will disincentivize spamming (probably not eliminate it) enough that self-hosting emails might be possible again without having to swim against the tide.

> So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers.

A MX records don't have to point to an IP; it can point to a host name.

My MX record is a dynamic DNS host name.

> Big email servers permanently blacklist whole IP blocks and delete their emails without processing or without notice. Some of those blacklists are public, some are none.

OK, but if you're having trouble sending, that's no reason to do anything with your MX record, which is for receive only. Just route outbound SMTP through someone forwarding service.

I've run my mail domain for twelve years. In that time, I've not sent SMTP directly to anyone; always through the SMTP forwarding host run by my ISP.

Well, you know, the mail is going through that ISP anyway! If I could directly connect to port 25 of various hosts around the net, I would still be routing through that ISP's hardware. So the fact that mail is routed at a higher semantic level through their SMTP server, rather than just at the IP level, just almost just a footnote.

I started hosting my own email in 2004 before finally giving up and migrating my email to Fastmail last year.

Besides the problems mentioned in this post the real problem I had was dealing with spam. The open source community around spam has really degraded over time, to the point where most solutions are extremely high maintenance and require regular tweaking. Methods that used to work, like greylisting, cause problems when dealing with GMail because google doesn't play nicely with it. The big spam blacklists have also gotten a bit less trustworthy over the years.

Is there actually a "big tech" email provider that accepts a message with a 2xx SMTP code and then deletes it? The only one I personally know of never does that. That one also does not use anything like an IP address blacklist. This article doesn't name names, it just waves its hands and throws around some innuendo. But as far as my own personal experience goes, this author has no idea what they are talking about.

The address 929@homeaffairs.gov.au, which must be used by Australian permanent residents to update their personal details, refuses to accept email messages unless they are from big tech.

Shame on you, Australian Department of Home Affairs.

And shame on Telstra, which provides the service.

---

Remote-MTA: dns; dibp-ibmail2.msng.telstra.com.au

Diagnostic-Code: smtp; 554-mx.msng.telstra.com.au 554 Your access to this mail system has been rejected due to the sending MTA's poor reputation. If you believe that this failure is in error, please contact the intended recipient via alternate means.

E-mail is complicated, sure. But I’ve had it up to here with people who give up running their own server and then go on to vastly exaggerate how infeasible it is, in order to placate their own conscience. It’s not that they’ve gotten tired of doing it, oh no; (they say,) it’s entirely the fault of Google, Microsoft, etc. who’ve made it literally impossible to run your own e-mail server. Except it’s not impossible – lots of us do it, still. And now there’s one fewer of us, so the rest of us have to work that much harder when the next monopolizing standard comes along (BIMI, anyone?). Sure, you don’t owe us anything, but thanks for nothing when making these public rants; you are scaring away people who might still be inclined to help!

The fix here is use a commercial provider for outbound smtp but continue to self-host inbound.

Not ideal, but it works.

In fairness, things like postfix usually ship with very poor (not to say “moronic”) defaults.

Like, postfix won’t even try to connect to tls-enabled smtp for outgoing email by default, and you have to explicitly point it at the certificate bundle it’s supposed to consider valid.

And you have to tell explicitly to reject incoming plaintext connections from the public internet.

And quite a bit more… Like, why doesn’t postfix have its own freaking spf/dkim implementation BUILT IN?

We still are fighting the oligopoly with our vaulted c1.fi email service. Please feel free to check us at https://c1.fi/v/hn20220905/?lang=en.

Here's EU's JRC-MECSA report on our service: https://mecsa.jrc.ec.europa.eu/en/finderRequest/b5daceffc76e....

Support for client's own domain is currently under works. Our webmail supports PGP and one can use IMAPS/SMTPS or ActiveSync based native email clients too.

All servers self hosted (we run C1/Gentoo!) in our own computing facility in Finland. =)

This isn't actually that hard to fix, it's just that for whatever reason, we seem to frequently have this blindspot that we don't seem to have in other industries.

Namely that "do it yourself at home" and "massive oligopolist" aren't the only two options. It's like saying "You can only have hamburgers two ways, cook them yourself or McDonalds."

I do the third and it's been great. I let my paid webhost handle it. (hostdime if you're interested, but I'm sure others do it well also)

Have around 100 users on my self-hosted mailserver. Works alright for the most part. Once or twice a year, there are connection issues to small companies with weird settings. I just route those over an external ESP.

Then there is also mxroute.com, which is an indie email provider. He seems to do fine too. Didn't use them yet.

So I think having at least some sending volume is key to running an indie server. You can't do it just for a few mailboxes/users.

I still wouldn't recommend to learn or start with email in 2022. There are better uses of your time.

> ... [They use] spam as a scapegoat to nerf deliverability and stifle competition.

Disagree. It was a way too open protocol to begin with. From a time of innocence best suited for places with inherit trust like inside a business. And it's not just spam. Phishing is also a huge issue.

As much as I want to sympathise, Email for the big WWW is unsalvageable IMO. Too many bad actors are out there.

> [Solution:...] * There should be a recourse for legitimate servers

This is the same Big tech story. They want to cut cost, you want a human touch. You can see similar stories here in HN every week. Which is why I think it will never happen.

The topic often comes up. Can't say I share the experience. My servers have never been put on a blacklist in the 7 years they've been running, and one of them operates from my residential DSL connection. Standard postfix+dovecot stack on an Archlinux VPS, I log in once a year to update the packages and make sure there is enough disk space left.

I relate to this. I also stopped hosting my own mail server for this exact reason.

However I do think it’s a case of damned if you do damned if you don’t. As a consumer of big tech email I become equally frustrated when spam makes it past the filter and I expect them to do more.

If it’s easy for the average person to setup a mail sever with high reputation then it’s easy for spammers to do the same. I can’t think of a great way to manage this at scale for the average person using a $5 a month Digital Ocean VPS sending < 10 emails a month.

One thing I have noticed is that there’s still a load of large organisations failing to implement basic deliverability best practices like SPF records. These organisations have themselves to blame.

After 19+ years of hosting my own email - It's worth it!

Imagine someone revokes your access or deletes all your emails because of an error, at the scale of gmail or outlook.com it just happens.

For spam there is one solution:

- implement greylisting. It just solves the problem.

Part of the problem is that the wrong people are complaining.

Using Google as an example, the author has no right to push anything to a gmail inbox. Google has no contract with the author to accept mail from him.

What Google is doing, it's failing its customers, the people who signed on gmail to have an address where other people could send data to.

And now those people are not receiving everything they could, but it's only up to them to decide whether this is actually a problem and whether it's serious enough to contact gmail support.

I do understand the point and the spirit of the author, but he is actually conflating the freedom of speech with the right to be listened to.

Hosting an email server on a consumer IP does seem to a losing proposition,

Hosting an email server on a cheap (reputable) cloud server and doing the basics (PTR records, SPF etc) still works well.

Very interesting, I've self hosted my e-mail since 1999, which incidentally is 23 years ago. My current server is at Hetzner in Germany.

Totally naive questions:

Could we come up with a new protocol (possibly based on SMTP/IMAP/whatever), that would only guarantee to get your email to its recipient if you included some sort of token generated by the recipient and given to you? Something where you could text/message/whatever a unique token to a friend/business/etc. and then they can send you email? And if you email someone, your outgoing email includes the token necessary for them to reply? The contents (including who it’s being sent to) would be encrypted by default rather than being plain text that anyone in between sender and recipient (or at least sender and recipient servers) could read. Is something like that possible?

Obviously at first nobody would have it implemented, so you’d have to get developers interested in writing server and client software, and convince people and companies to use it instead of or in addition to regular email. But I wonder how many people would be interested in such a system and whether it would be workable?

I gave up self-hosting ages ago. For a while I even used the old Google for Families grandfathred in setup. That also went the way of the dodo a few years ago for me.

I signed up for ImprovMX, setup my domains there, and just route my emails to whatever service I want. I use a random gmail account that I use for my login and Google services, but the email itself is never exposed anywhere, I only give out the custom domain one.

ImproveMX handles routing for my whole family. My mom uses Outlook, and it routes there for her. If google, microsoft, or whatever give me trouble or ban me, I just quickly switch the email and nothing lost.

If you pay for their service, since they have a super generous free tier, you also get SMTP servers to use as outbound, which lets me send emails through them and not have the `on behalf of` email thing. Also they do all the work to make sure their IPs aren't blocked and in good standing with MS, Google, etc.

How do people that work for Google on HN continue to work there without their conscience bothering them? These are terrible monopoly abuses and you’re contributing.

It should be possible for everyone to participate in internet systems equally as peers.

Somehow it seems like the Overton window has shifted such that people find it acceptable that ordinary individuals can no longer take part in the email infrastructure as equal peers.

I've been debating whether to do this myself (throw in the towel on mail). I am getting tired of fighting all the assorted battles.

This said, my concern is that the big players seem like they could, at a whim, drop you as a customer, with no recourse. This is what is giving me pause going to the big providers.

I've been looking at mail distributions like Mail-in-a-box, and modoba as an intermediate, though none of them seem to be great. Basically I don't want to stitch together several different opinionated tools into a working mail system anymore.

Funny thing happened to me today: Gmail sent its own Google Fi customer support email to spam. Haha, wish I noticed that before spending my morning going in circles with chat support.

I know of some small servers that get a lot of spam and hacking attempts, and their most effective tool against abuse is an IPv4 block ban. Increasingly this became more and more difficult, and I assume email servers are at the same point. Thanks to VPNs, people appear to be able to spawn insane numbers of random IPs.

One solution this decentralized server system came up with is the concept of accounts that have some barrier to entry to create (which involves a delay and proving identity). This account has a private key and it uses this to access the servers through any IP. Abuse on this account and any connected accounts of course leads to the key being temporarily revoked. Lots of positive interactions with well established accounts increases your credibility. Lots of reports decreases you credibility.

If you have been sending credible emails with multiple hosts for 10 years, even if you did get flagged, you would be given the benefit of the doubt. Hell, it should be easy to email the host and give them the headers and the reason why the email was flagged.

About the email space now being owned by big tech, it could simply be time for a boycott until they improve their practices. There is far too much centralization on the web now, and we all contribute to it every time we use an external service rather than host our own.

I am still hosting my emails via docker mailserver[1] I got some trouble with outlook.com bans, but Linode helped me to switch to a “good” ip address and it is working fine for now.

I will try ti resist as much as possible, because email is your primary identity “link” on the Internet, and you deserves to own it if you want.

[1]: https://gioorgi.com/2020/mail-server-on-docker/

Are the big tech email servers truly not used for spam at all? Because if they are, shouldn't they be permabanning each other as hard as they ban the little guy?

Clearly they're using different rules against each other than against small email servers, and I think that's all the evidence you need to get the EU to take action here.

My fear is that something similar will slowly happen to everything "compute". How long before my bank's website won't let me login if I don't use a computer with secure boot and a browser installed from an app store? McDonalds app on Android won't run on a de-googled or rooted device... At this point one may argue that there will always be computers that you can compile and install your own Linux. Yes, that is true. But just like I am not likely to have un-googled andorid for some apps and googled one for others, the same way it won't be practical to have one computer for some apps and the other one for others. And the one that will win will be the one that lets you login into your bank account, for simple and practical reasons.

I agree with the pains, but the options are not juts Big Tech or self-hosting. There's a myriad of not-big-tech email providers out there, for example there's Posteo, who use open source software and green energy. They are going strong for 13 years now with 400+k accounts.

https://en.wikipedia.org/wiki/Posteo

Reminds me of this fun story:

A person at a company mistakenly created an email list segment (or lack thereof) resulting in an email to the entire email list of hundred of thousands of emails. This combined with inexistent (we were a naive startup without an email specialist role) list hygiene practices meant we were blacklisted by Gmail after some time.

Took a year to get a hold of someone on Gmail's spam team. We found out were on 4+ Gmail blocklists, some of which were ML-based. We couldn't do anything to remove ourselves after we fixed the issues. A $1-2 million revenue channel dried up because we couldn't get out of the Gmail blackhole (short of rebranding completely, rewriting content, and using a different ESP). Fun times.

This thread is quite enlightening. It seems the engineering community predominantly would like to be able to self host email (and presumably other services). By proxy of that I guess that email hosting is not just for one user but maybe also friends and family or some community that you are part of. It's quite clear that managed centralised services accelerated the adoption of the internet, email, etc and the majority of consumers don't really care about self hosting but it also speaks to the fact that we've given up ownership of this aspect of our lives without clear understanding of how it will affect us in the long term. If no one but large corporations own all the services we use then it puts us in a pretty precarious position.

Know what will kill self-hosted email? Giving up and hosting your email with the big guys, the less self hosted email servers there are the less the big guys feel any requirement to support it

I am still doing it, on a cheap VPS no less. Yes, it is hard, and yes, some large email vendors drop my emails for no reason. However, if everyone throws in the towel, they (the monopolies) won.

Oddly enough I do not have these problems after self-hosting for close to 27 years now (i.e. from before spam became a problem). I hardly ever get any spam and my messages seem to arrive where they need to be even when that destination lies in Microsoft- or Google-land. I have the usual assortment of DKIM/SPF configured for my domain, I send mail through a smart host operated by my IAP (at no extra charge) but for the rest I do not do anything special. Am I the exception to the rule, am I just lucky that my IAP's smart host has not been blacklisted by the likes of Microsoft and Google or is the perception of self-hosting mail to be fraught with problems erroneous? I suspect the latter to be true, self-hosting is neither difficult nor bound to fail just as long as a) you have some good spam filters (easy), b) your MTA is set up with the correct SPF and DKIM records (also easy) and c) you send outgoing mail through a smart host (easy to configure once you've found one).

Hard to untangle the incentive for collusion + need to police legit bad behavior.

This could be a few large providers saying 'we control most email traffic, let's control all email traffic'. Or it could be serious players saying 'spam hurts our users, let's stop criminals using a blanket rule'.

More likely it's a schelling point where large players are rent-seeking (crowding out some competition), but only to the extent they can preserve the illusion this is about policing spam.

Suspect we'll start asking platforms to offer something like due process in the law -- administrative checks that increase the cost to administer a system, and reduce the quality to end users, but increase transparency and make it harder for the platform to engage in corruption.

"Newsletters from my alumni organization go to spam. Medical appointments from my doctor who has a self-hosted server with a patient intranet go to spam. Important withdrawal alerts from my bank go to spam. Purchase receipts from e-commerces go to spam. Email notifications to users of my company's SaaS go to spam."

Are you talking here about incoming emails? I expected that these would be reliably delivered to you and that the problem is only with emails you send to the large providers?

OP Carlos Fenollosa mentioned this:

> Over time I realized that residential IP blocks were banned on most servers. > You just cannot create another first-class node of this network. > Email is now an oligopoly, a service gatekept by a few big companies which does not follow the principles of net neutrality.

It's unfortunately true. However, the reason that how we end up like this is more nuanced than just the big players trying to power grab (perhaps) but rather because of the rise of spam/scams/phishing/malware. All big players like Google (Gmail), Microsoft(Outlook/Live.com/Hotmail), Yahoo!, Apple (iCloud) are suffering from those threats, wasted bandwidth and compute on spam detection heuristic AI.

There are industry consortiums like Spamhause and commercial entities like Barracuda to maintain blacklist/whitelist to restrict access of major MTA network interconnect to fight off spams/malwares/phishing/malware delivery from botnets and individuals. And it helps, at the mean time, it consolidated the control of who can send outbound emails.

We are seeing this trend repeatedly in other communication channels like phone calls (due to robocalls, VoIP numbers are being blacklisted by all major players' services) or Text messaging (due to spam texts, major U.S. wireless carriers band together established Campaign Registry to control who can mass send outbound text messages. This is also known as 10DLC registration).

I think the vulnerabilities of previous communication protocols (email, VoIP, SMS/MMS) lie in the fact those protocols are designed with security in mind. Modern community protocols like Push Notification has been designed with security in mind, which make it less susceptible to abuse and spamming. That's probably the way go forward.

My little e-mail Server on an OVH VPS is happily sending and receiving e-Mails to/from the big ones without problems for my 20+ domains. Just a basic postfix/dovecot setup with letsencrypt certificates and SPF/DKIM/DMARC working the way it should. I described everything in a short blog series at https://jan.wildeboer.net/2022/08/Email-0-The-Journey-2022/ in case you are interested.

Yeah, I ran my own e-mail server from 1995 until about 10 years ago. It used to be fairly simple, but between incoming spam and attacks, the various standards that were developed, and my e-mail not getting through... this became a problem that was worth paying someone a few dollars a month to solve for me.

I still use my own domain, but I'll let actual delivery and security experts deal with the day-to-day running of things, while I run my business (which definitely isn't that).

It was a bit sad to give up, but the time and frustration it saved have been more than worthwhile.

Proper, artisanal self-hosting of email can still be viable depending on your expectations and tolerance levels for random issues.

These days, I operate with the medium-temperature bowl of porridge: AWS WorkMail with custom domains & users. My use case is basically "Replace gmail for personal email". I don't have a lot of patience for running an actual email server, so this is about as custom as I can get.

Running a custom email domain can have other practical implications, such as having to carefully re-iterate spelling when mentioning your email address over the phone to a customer support agent. With a gmail or hotmail account, virtually everyone can type that hostname in without thinking about it. This concern is moderated by being able to select a username with fewer than 5 characters, rather than your full legal name appended with your date of birth.

The cloud services providers have some of the worst IP blocks for spam. Cheap hosting blocks are a distant second place.

Google for example, doesn't even hide the fact you have to request whitelisting using their online business services portal. They don't give a toss what kind of sender authentication/signing hoops you have jumped through already, and user letters may still end up in the spam bin.

Many users have indeed migrated to the web platforms, and don't care about people data-mining their business communications. The real issue is so have many spammers/scams, a side-effect keeping their trade on life support by removing technological administrative barriers for the desperate... and you can't block Google/Outlook/Redmond.

Completely agree this is anticompetitive and worthly of examination under the light of antitrust laws. Of course the defense that will be offered and one which is partially true will be one of providing safety from spam, phishing, and other evils prevalent in the email system today. Maybe texting is the new email after all...

The real problem is and always has been "reputation". Now the big boys fuck that up by definition from the start and require you to do the heavy lifting: DNS, SPF, DMARC, DKIM and so on.

I run several email domains quite happily in the UK. I know why it works and I don't resort to magical thinking. My ISP is considered a business one and my IPs are static. I've owned both my work and personal ranges for a while.

Feel free to contact: furtle@blueloop.net - I'd love to hear your ideas.

Cheers Jon

I've been self-hosting since 2004. I currently route via a VPS. The only issues I have seem to be with outlook.com/hotmail.com/etc - free Microsoft accounts. That goes to junk, though replies seem to work fine. Paid Outlook365 seems fine.

Even after speaking with Microsoft's email admin team on the phone a couple times, I still have issues. It's kind of infuriating.

I have properly configured SPF+DKIM (selector rotated daily)+DMARC, and I've gotten set up with dnswl.org.

I hosted my mailman stack on VPS for some time, it worked well

I stopped self-hosting because it's too much hassle, but it was any difficult to maintain, (by difficult I mean complex)

It didn't worth the time I spent though, so I quit, but I would do it again if I need to

If I had to maintain a server at home and my ISP blocks it, I would get a VPS and host proxies on the VPS and use VPN tunnel to keep the mails stored locally

But I don't have any reason to do that currently, as well as most people

I already wrote this also in another thread. Which was not also only was suggested by me:

* implement and publish policies which emails you accept (regex/strings on domains, emails, headers, signatures and so on)

* found a association where all use this strict (which shall potentially be stricter than gmail and so on) settings and if gmail does not accept these emails, sue them for discrimination.

I know a lot of older guys will disagree with me here, the guys that use mailing lists to work on FOSS projects, many of whom I respect very much, but I think email sucks. I only use it for the same reason I have a phone number: because some people need me to have one in order to contact me.

I don't like email. I think the problems with it are shortcomings of the protocol. I'd rather not use it. But I do, as a last resort contact method.

For me, people that use email are like people that primarily communicate over SMS. If I need to talk to you and that's all you'll use, I can. But if there's another way to talk to you I'd rather use that. Xmpp, matrix, signal, shit even telegram and discord if I have to, are preferable to SMS or email. But otherwise, yeah I have some email addresses and a phone number if you insist on doing things that way.

Author do not mention the frictions of going the other side (gmail/hotmail/live):

- you might not receive all the emails people sent to you, for the reasons mentionned in this very article

- you will receive much more spam/unwanted email. I quit using a gmail account because I kept reveiving newsletter and notifications from other people who kept mistaking his account with mine. There was probably a one letter difference in their real email address and mine. I surrendered and gave up trying to tell his relative they weren't reaching the right person, only had fun once by powning his NAS with a "cloud function", and gave up after receiving tons of newsletters and other shit.

- you can lose your account any day, for any reason without any possibility to get it back. I've seen it happen to 2 people with hotmail.

Of course it all started with spam. But the response was just as destructive. I worked for a small ISP (25K email customers) back in the mid-2000s, and it was so hard to keep email flowing. All it took was one random customer somewhere on our DSL who had a misconfigured open relay, and spamhaus would blacklist us. They wouldn't respond when we'd try to get the block lifted. We tried and tried to work with them to streamline the process, nobody wanted spam to go through our servers, but they weren't really interested in any cooperation. They were perfectly happy to stop all legitimate email if it stopped a single spam.

I don't work there any more, but I'd be surprised if that little ISP hosts their own email servers nowadays. It's so expensive to deal with such issues, it's just not worth it.

It’s time for the US mail service to digitise and fulfill its constitutional obligation to provide a private, secure mail service for all American citizens.

How I wish spam was handled:

1. Any email that isn't signed with DKIM is blocked (having a signature specific to the source email address would be better, but that is probably too much to ask for)

2. If the sender isn't in your contact list, block it, or at least mark it as spam.

3. Have an easy way to add new entries to you contact list, maybe a new url scheme similar to mailto:? So that it reduces the friction of say getting a confirmation email when signing up for something, or making sure you get emails from a new acquaintance. It would probably be good to have a way to add a full domain to the allowlist as well.

But that would make selling/sharing email addresses a lot less valuable, so there might be some resistance to that from marketing and adtech.

> Blacklists should not include whole IP blocks. I am not responsible for what my IP neighbor is doing with their server.

This is obviously laughably naive and creates infinite sources of spam.

Before doing a proposal on a core Internet technology you should be required to be on the other side for a while. Do anti-spam at a large retail e-mail service provider for a year and then you can understand the problem space.

You might not be responsible for what your neighbor is doing with their server, but the ESP is responsible for filtering it. The idea that they need to treat each and every comcast IP with equal weight is nuts. IP reputation is the single most valuable tool in the industry; the largest statistical predictor of whether or not an email is abusive.

I have self hosted for 20 years. If you want perfect delivery then completely self-hosting isn't for you. It is a good learning experience and once setup it does not require much attention.

A compromise solution is to outsource delivery which is by far the shittiest part of self hosting because of bad business practices and lack of regulation. It is the least interesting bit anyway.

My email server is setup with policies commented out to send outbound emails through another host I maintain if required. When a very large company hosting lots of email for many domains mass ip blacklisted my hosting provider late last year I used this to maintain connectivity while the companies sorted out their dispute.

It's strange that in his list of possible solutions to keep e-mail spam free, there is zero mentioning of hashcash [1].

No, that is not a shitty crypto coin. It's just a computational proof that your computer spend some seconds on hashing, which is fine if you send 1-to-1 emails like real people do, but not if you are a spammer who bought a file with 6 million leaked e-mail addresses.

See Back's 2002 paper "Hashcash - A Denial of Service Counter-Measure".[2]

  [1] https://wikiless.org/wiki/Hashcash

  [2] http://www.hashcash.org/papers/hashcash.pdf

This is pretty much exactly the reason I stopped running my own mail. I had the mail host on a static IP, part of a block of "pristine" IPs from a local colocation operation (iNOC in Albany, NY). Ran fine for years, but I started having to call more and more customers about "why haven't you answered my email from two weeks ago?" only to find out I ended up in the spam bin. Wasn't worth it when the risk was losing business.

I moved to Proton Mail as I like their simple interface and support their goals. Pretty good service so far, worth paying for, but I do sort of miss running my own services.

If you run your own mail server in 2022, you are the resistance.

I host my mail for around 12 years so only half as long as the author of the article, I faced the same problem- that my emails land in spam for some people. In most cases it's those people who care to receive my email and I always tell them it's their provider who is at wrong and that they should switch. We have some laugh together, exchange some jokes and continue with our lives. I'm one of not many who love postfix and dovecot enough to use it to self host, but I'm fine with that, I won't throw towel and will continue to run my email, hopefully I'll be lucky enough to run it for next 12 years :) Peace everyone :)

I'll keep going with my private server (on a vps). I just moved it to another hoster, meaning new IP block. This caused blocking issues: Some but not all gmail address delivered to the spam folder. Another big mail provider asked me to set up a web page with contact info on the same domain I use for mail. And another self hosted mail server from a public service agency had me on block list.

This caused me some headaches and I was thinking this could be the end and I have to use one of the big players. But I did not give up, invested time and it works now again!

I am self hosting. It works and I have no problems.

I use https://cloudron.io for orchestration, security - to run it on a VPS. Everything just works.

I did training provided by a large email security firm, and one thing the presenter said was along the lines of "this spam filter defaults to block the senders domain & IP, you can set an expiration on that block butI don't see a reason why you would". One misconfigured server sending out a single email and I assume by extension someone impersonating your domain could get you perma-blocked from sending emails to that company, and I assume it'd reduce your trust rating for other orgs using that provider.

to me, the root cause of this is just money. a LOT of people have zero scruples when it comes to money, so they consider sending email to every email address they can find, in order to hopefully make a sale, to be a perfectly valid tradeoff.

these people deny that they are causing a problem, or that they ever caused a problem, because admitting that would mean they are a bad person, and they're not a bad person! they're "just trying to feed [their] kids, man."

making decisions based on money alone is always a bad idea. ALWAYS. I do not care if it is one person and one decision, or if it is a business making a decision on behalf of their stock holders, or anything else.

if money alone is your decision-making criteria, you are making a bad decision, or you are making a decision on bad criteria.

someone always pays for everyone's scramble for money. someone always pays, and it is always an unjust payment.

in this case, spammers have cost us our ability to self-host email, which is a very significant problem, as described by the author of the linked article, with rather severe consequences if you hope to have any freedom on the internet.

so, if you work for a company that will, over time, do just about anything to get people to click on ads, you are slowly destroying the internet as it was intended to be, and was, for 2-4 decades, depending on how you define that.

I self host my private email server since 2005. Never had big problems. One time my server was wrong configured and then marked on spamhaus. Just fill in a form and all works fine again. Maybee gmail will blacklist me in some weeks, but fuck you gmail, I have nearby no one that uses a gmail account. I had in all this year no problem with any other peer. So, maybee the writer has just a malconfigured mailserver or don't know that he is marked on spamhaus because any reason.

The range IP bans hit home. Been hit by this more than a few times. You don’t have to be sending spam, but if someone in the range did, ever, you will be blacklisted all the same, and treated like a spammer.

The people who run these blacklists are unreasonable. I can understand why, they tend to interact with the bowels of the internet and the heuristic is effective. Usually people who need to talk to them are doing something naughty, so why bother taking a chance?

Guilty until proven innocent would be an improvement.

1999 was a pretty late adopter to the self-hosting email cause since it was pretty apparent since 1997 that free email was going to win out. As Gmail filters get smarter then it becomes an altogether more fruitless cause. Email hosting at the individual level died out for most due to this and the pointless rivers of spam you had to deal with. If you've only now come to the realization that you can't make it then you've been wasting a lot of your life.

I've been hosting my email on my own server since the 90s, but got tired of dealing with keeping up with spam filters. Updated the MX to deliver inbound to mailroute.net and have them do the filtering before forwarding to my server and that's been working great for years. Not free, but not expensive and still gives me 99% of the control i want with very good spam filtering too.

Outbound mail is relayed via mailroute too, which solves the tainted IP delivery problem.

Sending emails with dkim + sfp and I never had big problems with reachability and a postfix server on Digital Ocean.

I haven't done it since last year though. Has something gone terribly wrong?

I remember debugging issues with email sent via aws ses to Hotmail addresses at $dailyJob but I can't think of a single Microsoft product that works well (windows, teams, azure, now even GitHub is starting to work every other day) so it doesn't surprise me.

Big email companies were never threatened by self-host. Spammers ruined it, not big email companies. Spam is an incredibly difficult problem to solve.

I managed 3+Mail at 3Com 35 years ago, and in fact, it gets its own subplot in The Big Bucks (https://www.albertcory.io/the-big-bucks), back when email was brand-new (well, for most people).

However, nowadays I'm bored with stuff like this. PITA. So I totally sympathize with the author.

My company exists to solve this problem at scale for the web hosting industry. It’s too bad that self-hosting isn’t viable because of IP reputation problems, but it’s a reality that is unlikely to change any time soon.

I’d say if you want to continue self-hosting, just let go of the delivery part. Use a service like SendGrid; it probably won’t cost you anything and it’s easy to set up.

I've been trying for the past week or so to break into the self-hosted email game since I had a VPS and found some tutorials and documentation for postfix + dovecot. I did eventually get inbound and outbound email working although I was using an alias when I think I needed to use a virtual mailbox (to both receive and send email from a user with a different name from the unix user behind it, as right now outbound still leaks my unix username).

But then I looked at Protonmail's cost and it's less than I'm paying for my VPS (which is already cheap for the use I get out of it) so I'm on the fence whether I keep hammering away at that (and then have to wrestle with the big players treating me like a spammer and do my own spam filtering) vs just pay for that convenience. The VPS is staying in any case so it's just a question of whether I pay a little more every month for convenient secure mail.

G'Day Carlos We started hosting own SMTP servers on IBM OS2 using a BBS. Long before retail NET was in Oz. Fast forward to today and we have a DB with millions of IPV4 address and have never used IPV6, cloudflare or a 3rd party to 100% stop spam like we do day in day out. You are absolutely right, there are a lot of criminals you are wearing suits who are selling snake oil. We have X simple rules, one is that inbound port 25 sessions from spammers are put to sleep and we give them a very poor service. Our servers seem overworked and they often go away never to return. I read your blog post and from approx. 30 years experience including SMTP at gov.au and gov.nz content filtering does not work. Our NoSpamAccepted ( NSA ) AI powered SMTP servers use the network to stop spam. How many domains & sub-domains are you self hosting ? TIA KG BTW - We have never been in the BLs as we only relay our outbound

I have been self hosting email for just over 23 years and I am more emboldened than ever to keep doing so. Even with SPF/DKIM/DMARC setup, I am constantly asking people to check their spam folder, add me to their contacts, etc. I refuse to pay an email tax to one of the larger players to solidify their hold over the protocols.

Did you properly set up DKIm, Dmarc and SPF? I have been hosting my own email through a VM in Texas for about two decades. Whenever I had problems with rejected deliveries, they were solved by moving to the new standards the internet invented to fight spam. Did you use a dmarc analyzer to verify you were set up correctly?

I self-host email for a bit less than 20 years and still not given up. Being blackholed by Microsoft or Google is a real risk (though I was lucky so far), but at least I fully control receiving side - if I expect a message I can check my logs to see if there was a delivery attempt if it was not successful why.

I tried to host a home email server a few times and it was a pain to say the least. Finally I created a droplet on Digital Ocean and used Mail in a Box https://mailinabox.email/ with a glue record to act as a name server. So far so good

I like a lot of what’s being discussed here. One thing to consider is a similar problem I am facing is people trying to hack into a login page. We see thousands of requests per minute from different IPs… many from VPS, some from obviously hacked TVs/devices. We could implement an exponential back off on abusive IPs but detection requires observation of action that action could result in a compromised account… so another idea is we simply block large ranges of know bad IPs from blacklists… I think this is similar to the email sending issue… it’s not fair and I think a solution could be some kind of “block chain” - make it expensive to login… make it expensive to send an email… but I’m not sure and for email it’s way harder because you need agreement from the oligopoly of email providers… not sure what the solution is

There is even a much larger problem here other than not being able to self-host your own email. The issue is that large providers and large companies have no accountability. No way to ever have a conversation with an actual person (email or by phone) to resolve a simple problem or mistake in an algorithm or automated process (or even a manual process if that's what it is). Canned replies not reply to follow up question not a care in the world or a conscience. And the fact that the service is free (to someone) does not mean a company should be able to so easily do what they want and cause aggravation to others.

And it happens even with paid services at large companies.

That is not 'just business' and has never been the way business operated pre-internet except in a few super rare (and perhaps rare monopoly) situations.

The Helm email server — funded by future YC CEO Garry Tan’s VC firm, but I bought one before he returned – is a really great compromise between privacy and convenience.

The IP block is managed by the Helm co., they tunnel connections and sell you the (tiny, silent) server and software. Each Helm server generates its own TLS cert, so the tunneling does not violate your privacy (unless it was delivered without TLS, in which case your privacy already vanished upstream).

The only delivery issues I hit are sometimes with Outlook/Microsoft managed domains. It’s been at least a year since I had that issue. When I first bought one someone on gmail had to move a message of mine out of spam, but it’s been fine since. Last I checked their infra is hosted on AWS but apparently they have some screening technique for getting clean IPs.

The general rule of thumb for my home server is that messages to people I've contacted before, or to addressees that were in a thread that I'm also part of get delivered reliably. Messages to new people I've never emailed before often go to spam. I've learned to accept this.

Not to mention that your email is tied to your online identity in many ways when mostly every account you make asks for an email to use as your username, verify your account, or recover access to your lost account.

Handing over control of your online identity like that to a centralized third party when you could be cut off at any point [insert here any recent news about people wrongfully losing access to their Gmail account and not getting it reinstated] seems like the wrong solution to be allowed the privilege of sending email.

I guess the very least we could do to keep some control is to own (read: rent) your email domain you could move elsewhere in case you lose access, but then you gotta make sure you don't also lose access to your email domain.

Is there an actual solution to all of this?

Giving large providers a taste of their own medicine might be interesting, let gmail spend 24 hours on a black hole list, make sure the removal forms have a captcha so they can’t be filled out by automation, when people complain on social media then let people know what it’s about. Social media really gets Google’s attention, it draws them out from behind their algorithms and silence when nothing else will.

Utilizing IPv6 more might help also since at some point it becomes absurd to have quintillions of addresses blocked for all eternity.

There are monopoly and racketeering angles also, since Google and Outlook are suppressing independent mailers in favor of their own paid products. Nobody will make any money except the lawyers but that’s ok as long as the situation improves.

A big problem with the way the big companies fight spam is that sometimes it is a bit too aggressive. Happened a couple of times where legitimate government-agency emails had trouble getting through.

Personally, I prefer defining my own spam list rather having an algo decide what pops into my inbox.

What do I do? I got rid of Spamassassin after being challenged by someone on HN or Reddit, can't recall which. Spamassassin is pretty effective but was consuming too much of my time.

So I stopped. I analyzed mail for a few weeks to look for patterns in the wild with my server. I came to the conclusion to block all but the top level TLD's. That decision yielded very positive results.

I then wrote a simple SPAM blocking server to allow me to block habitual or suspicious TLD SPAM sending domains, as well as a few custom checks for common sense things.

As a result of those two decisions, I am now at or better at blocking SPAM than I was with Spamassassin and 2 other related tools I just remembered I also stopped using, spamass-milter and postgrey.

RBLs and other blacklists have been annoying for decades, even more so when they were maintained by annoyed individuals rather than corporations like today. You had to beg a single person to remove you off a list that an ISP somewhere used without thinking twice about it.

Honestly wish there were a mode where you pay me $x to send me email and I can sort by spend.

>In many countries politicians are forced to deploy their own email servers for security and confidentiality reasons. We only need one politician's emails not delivered due to poorly implemented or arbitrary hellbans and this will be a hot button issue.

"I just the other day got... an Internet was sent by my staff at 10 o'clock in the morning on Friday. I got it yesterday!", as Ted Stevens would say.

Unfortunately the man said this as part of a massive, uninformed speech[0] about why big tech[1] needs less regulation.

[0] https://en.wikipedia.org/wiki/Series_of_tubes

[1] Comcast inclusive

I'd like to consider migrating from my self-hosted solution but it seems like all major email providers want $ per month per user|mailbox. Are there any providers out there that charge a reasonable fixed amount for say 50 mailboxes?

Same here, I’ve been hosting my email since 2007 and gave up in the beginning of this year, mostly due to constant delivery issues with microsoft infrastructure, which can be mitigated for a couple of months before coming back again

> The industry should fix email interoperability before politicians do. We will all win.

Not sure if by "politicians" he means legislators, but given very few players that control today's email deliverability, while doing very little to provide observability (=feeback loop) to the users who needs it most (that is users who cannot afford to build an expensive pipeline that optimize deliverability), given all that, I think regulation around distributed protocols observability/fairness is not unlike AI explainability regulation, only I expect that with mail it shouldn't be as hard to implement.

For those interested, I made a post related to this topic a few months ago: https://news.ycombinator.com/item?id=31180379

Yup. I have to pay Google a chunk of change every month to do something I used to be able to do for myself for free. Because suddenly they flagged my domain and no one would get any of my e-mails.

And there ain’t nothing I can do but pay.

I've been self-hosting my mail for 17 or 18 years now, by purchasing just some managed webhosting package from somebody who cares about their services not being used for shady stuff (any reputable managed hosting provider) and I think I've never "lost" an outgoing mail for personal use.

I don't understand what the author thinks it's so hard here and why he's painting it so black and white. There's lots of more to "my own e-mail" than choosing between some old notebook running and collecting dust in your garage and using GMail.

Some people just want to find a hair in the soup.

I've been using exchange online for my personal email. I had to do a bit of fenagling with my dns records, which is old hat at this point, to get it to work but it basically costs $4 per month per user and since I'm the only user it gives me the freedom to use my own domain name and not have to worry about selfhosting. Prior to that I was using exim as my SMTP server and dovecot as my imap/pop3. But managing it was a huge hassle. $4 a month is worth the effort of not maintaining it myself.

Totally unrelated. But this guy has an amazing OS-from-scratch tutorial [0].

[0] https://github.com/cfenollosa/os-tutorial

> Email is now an oligopoly, a service gate kept by a few big companies which does not follow the principles of net neutrality.

While I understand what they're getting at here, I disagree. There are certainly other providers you can go with - You just need to pay up.

Happy customer of FastMail here. All of my personal domains, and multiples businesses are tied to it. Wonderful service, no deliverability issues, great features, etc.

So - clearly other companies are able to get it right. Self-hosting is probably difficult, but it's not like you're forced to go to Big Tech.

I would just like to point out, you _don't_ actually own your domain...

It's ironic, 23 years for me too and my towel is not thrown in at all.

Just did a test with my own mailserver hosted on a small-ish local vps.

Outlook: OK (had to bother with this one when I started out) Google: OK iCloud: OK

I have a pristine track record and not a single byte of outgoing spam so I cannot attest how easy is it to get back into the game after an incident. I do agree with the larger point being made here. It is clear some kind of anti racketeering legislation would be the only fix. Sadly, currently there is zero will from both the EU and the US to fix any of these blatant anti competitive issue on the internet.

The dream is a self-hosting email, which is also super easy to setup, so much that any person in the world could do so without specific technical knowledge.

How? I don't know. But it would be great.

I just moved from fastmail to self hosted and it works with no email blocked so far. Had that setup for 6 months. The most important point being sure to have a static IP with reverse DNS.

I am running my own mail server for 3 years now. There were some issues (outlook pushing emails to spam folder) and an IP change (not sure how that affected things but I resubmitted the IP to the Microsoft email tools, can't remember the name of the service). I would like to join some movement or a class-action lawsuit against big tech for anti-competitive practices regarding private mail servers if such thing exist, and if not i would suggest we make one.

I love the idea of self-hosting my email, but there's no way I'm going through all that work if getting my mail delivered will be a toss-up. Complete motivation-killer.

> We are all experiencing what happened when politicians regulated the web. I hope you are enjoying your cookie modals; browsing the web in 2022 is an absolute hell.

What would they do with email?

To be honest, I kinda like seeing a lot of cookie modals out there. Yeah, the experience can be hellish, but it highlights how many sites are actually collecting data from their users.

With that said, I wonder what alternative regulations are feasible if we don’t rely on politician-mandated regulations.

I also came to the same conclusions 8 years ago after hosting my own email for more than 2 decades. You just can't do it on your own anymore. There are some proposals how to fix this, but I think a more sane approach is whole email system overhaul/replacement with something better. Many people are thinking about this and come up with good ideas. I hope eventually one of them will get enough traction to replace our frankly outdated email technology.

I have had the exact opposite experience, I used mail in a box and set it up on digital ocean on a 5$ droplet.

Have not had any spam or blacklisting issues and it was super easy to setup.

At what level of the net does IP address reputation operate? Do the blocklists ban whole ASNs based on some measured volume of spam?

More transparency (or more likely, less ignorance on my part) here would be helpful.

Can anyone recommend a one stop tool / script for looking oneself up in the reputation services? (And on that note, it is abhorrent that Big Email providers don’t have open reputation databases, or at least ones where I can look myself up.)

«So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers.»

I would have kept the MX records pointing to my personal server, and I would have changed only my configuration to send outgoing email through a third-party relay (eg. Gandi). This would have solved all the author's problems (deliverability issues) while staying 99% self-hosted.

Wrote my smtp server 5 years ago. Still running.

BTW, did you know the smtp protocol works without DNS?

You just need to puth the ipv4 between brackets @[xxx.xxx.xxx.xxx] and for ipv6 @[ipv6:...].

spam? simplicity and freedom has a price (personnaly, I have have very, very little spam since I am self-hosted), and don't think corpos won't try to force you to use their servers one way or another... Whose coding the virus? It is sane to presume it is the seller of anti-virus software...

I self-hosted my email during some years. My IP was banned in certain providers a few times, I guess due to neighbors. I also found quite complex to manage the service using dovecot at that time, but I guess today we have simpler servers.

However, I ended up moving away. Sadly, dealing with deny-lists and the management overhead was not worth for me. Fortunately, there are alternatives that doesn't require you to go to big providers :)

> We are all experiencing what happened when politicians regulated the web. I hope you are enjoying your cookie modals; browsing the web in 2022 is an absolute hell.

This. Just This.

Seriously though, as someone fairly uneducated in the space of how standards, protocols, and regulations get set in place, how can the ship be moved on this issue? Tech companies will likely only move / allocate resources if there is financial incentive. So what do we do?

A similar problem does exist when hosting your own crawler. No matter how polite your crawler is, after a few days your IP address gets blocked.

The suggestions are good. Do Postfix and other FOSS email servers implement them?

> - Let's keep antispam measures. ...

> - Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty. ...

> - Blacklists should not include whole IP blocks. ...

> - Stop blackholing. ...

> - There should be a recourse for legitimate servers. ... doing some paperwork or paying a fee to prove I'm legit ...

Nah. I'm glad that people like the OP are squeezed out of sending their own email. I have no problem with an email oligopoly at this point.

For every "good" email server owner, there's probably a million bad ones. And the problem of spam is a big one. If you want to send your own email, get used to telling people to check their spam lists and/or add your email account.

Not seeing anyone talking about email lists, now you need to buy a mailchimp type service, but many mailchimp type companies ban political customers if your not on their team.

Setup a private listserv or mailman use to be easy, but now you need to have a smtp provider in front, or you will quickly get blacklisted. Even then, get too big, and you will trigger some email email providers.

The competition argument is getting weaker and weaker, IMO. The point of competition, in theory, is to facilitate a kind of darwinian process that iterates toward an optimal solution. We have basically found that: Gmail pretty much just works for 99% of people. I sympathize with the 1% it doesn’t work for, but appeals to competition are not the way forward.

The domestic ISP I use has all its IP ranges assigned to spamhaus's Policy Block List. https://www.spamhaus.org/pbl/ However, the ISP does operate its own SMTP proxy that's free to use. So I can send and receive emails using home IP address.

This is sad, but it's a gross and inaccurate oversimplification. Let's look at the summary of "What are we left with?":

> You cannot set up a home email server.

This is true enough to not care about edge cases.

> You cannot set it up on a VPS.

This is definitely not true.

> You cannot set it up on your own datacenter.

This is absolutely, unambiguously untrue.

I get that there are many people out there who don't want to administer an email server, or who administer one (or more) and are tired of trying to train users to DTRT and care about security. The truth is that if you have lots of users, it's likely that one will get compromised, and their account will be used to send spam.

Is it the end of the world? Heck, no, unless you let it go on for days. "It's not if, it's when. Say goodbye to your email. Game over. No recourse." That's just plain not the case at all, unless, again, you don't have monitors in place.

A super simple example: a script which counts the number of email sent by any given user in a certain timeframe is really not complicated. I've used something like this and it has caught a mail loop which wouldn't end because the entity causing the looping was rewriting so much that typical anti-loop checks failed.

So a user gets compromised. If this is a real concern (say, for instance, you have a lot of Windows users), your script should send an alert to you when this user's account has sent several hundred messages over the past hour. You disable the user's account, you clean the mail queue, and you deal with the fallout. Sure, that may mean watching your logs for a few days for rejections and visiting other networks' delisting pages, but it happens.

So there's the largest problem with running your own email server handled. Boom. Done. If you've hosted email for years yet can't / won't do this little bit of work, then that's you. The rest of us understand this.

What about deliverability in general? Isn't that the largest problem, you ask? No. No, it isn't at all. You can even run an email server on your home Internet connection, if your ISP allows incoming connections, the same way you can handle any other general deliverability issue: smarthosting.

If you want to claim that there are NO ISPs out there that can reliably send email outside of Yahoo / Outlook / Google / Amazon, then you might say smarthosting isn't a solution. However, you'd be flatly wrong, so wrong you shouldn't be hosting email.

If your home network can't send email (it almost certainly can't), and your VPS can't send email (it'd probably have issues), and your datacenter can't send email (you're clearly doing something wrong, but let's pretend), then you can smarthost through an email provider that has a good reputation. Period.

Anyone who wants to argue that hosting your own server can't be done today because of deliverability ignores this super obvious solution, which negates this entire article.

Let's move past that and look at the suggestions this article makes:

Should we throw in the towel, proverbially speaking? Certainly not. I disagree with this emphatically.

"This doesn't only affect contrarian nerds." No, it doesn't, but discouraging others isn't the solution. Your lack of solutions isn't a good reason for others to throw in the towel. But why are so many "contrarian nerds" so quick to tell others to NOT do something? Do you tell people to not paint or draw, because it's too hard for you? Or to carve, or write fiction, because you're not good at those things?

"You can no longer set up postfix to manage transactional emails for your business. The emails just go to spam or disappear." Nope. You're accepting that as normal and equal. It isn't. This is the same basic idea as "I can't afford to not run Windows, because everyone else runs Windows" - it's a fundamental misunderstanding on your part that leads you to assume you're the victim, and you're powerless. If your email is being silently dropped, then you need to tell the recipients that they need to 1) complain to their provider, and / or 2) find real, deterministic email services. I've told many people that I'm not responsible for overzealous spam filtering, and I provide proof that the email was delivered. It's on them after that. "But I can't afford to do that!" Then smarthost. This isn't difficult.

"One strike and you're out. For the rest of your life." Nope. Demonstrably, nope, unless you're letting spam flow from your servers for days at a time.

Your recommendations:

"Let's keep antispam measures." Sure, but consider the fact that they're part of the problem. Spam filtering shouldn't be arbitrary - for instance, I do ZERO content filtering, unless or until I can prove to myself that there are no false positives. Email with "storage.googleapis.com" URLs? 100% spam. Email from random addresses / networks with Gmail Reply-To? Absolutely 100% spam. Email from servers with a HELO / EHLO name that doesn't exist? Rejected. But keywords? No. That's stupid. I've seen, for instance, too many abuse email addresses that don't accept spam complaints because of content-based, rather than behavior-based, spam filtering. The problem with Gmail is that they do too much content based filtering, with no rules and no logs that we can see.

"Change blacklisting protocols so they are not permanent and use an exponential cooldown penalty." Fair.

"Blacklists should not include whole IP blocks." I disagree. If your network neighbors are shitty, then you should 1) ask for your IPs to be SWIP'd to you, 2) find a better company that punishes spammers / scammers, and/or 3) smarthost.

"Stop blackholing." Yep. But, "No need to bounce every email" - 100% disagree. If you're sending so many messages that you're overwhelmed by returns, then you're doing something horribly wrong. Every email needs a bounce. This is how email works.

"There should be a recourse for legitimate servers." 100% agree. I think someone who has the time and resources should take all the large providers to court to compel them to have methods for correcting interoperability. If Google, for instance, wants to be like a utility, then they should be forced to act like one and they should have real ways to interoperate. As it is right now, it it not possible to reach an actual human at Google about anything via email. Every single message goes nowhere. They shouldn't be allowed to operate like that, or if they want to be arbitrary, they should lose the right to be called RFC compliant email and the use of Gmail accounts shouldn't be usable for anything public. That's another whole battle, though - why should a company get to call themselves an email provider when they don't provide reliable, repeatable service? Sigh.

"Email discrimination is not only unethical; it's a risk for the industry." Agreed. I think there's already legislation proposed, if not already passed, making certain types of communication unblockable. It's shitty legislation, but it's a first step at a precedent we all need - we need to be able to dictate to large corporations the parameters of what they can do and can't do if they want to be considered email.

What the author doesn’t mention is he uses his email to discuss his penis enlargement company with his friend, a Nigerian prince.

OP mentioned on Twitter that he chose iCloud Mail as the provider. How do You handle issues with losing incoming mails? Apple very often rejects mails from financial companies or ones that contain particular words and they won't report that to You. That's the only issue that keeps me from moving from own server.

So what do the companies with Exchange and Lotus do today? Do they still host their email-servers themselves, or already gave up and moved to big-providers? I can hardly imagine companies will leave their emails on Google or Microsoft for them to scan and abuse. But I'm so many years out of corporate environment.

A bit crazy to call this unethical. By that logic McDonalds is unethical for not building a restaurant in the middle of the amazon rainforest. Hey, I want my chicken tenders! Well tough cookies, friend, if there's no economical way to support a customer, said customer shall not be supported. Nothing personal. Nothing unethical.

Couldn’t agree more. I had to leave Zoho even though I didn’t want to. Google just blocked emails to my Gmail customers and left me with no choice.

Write about it here https://bitbytebit.substack.com/p/customer-hacquisition

Email is a protocol from the seventies (unmixed, unencrypted by default), not surprised if it leads to centralisation.

Maybe the approach is all wrong? Maybe we should be educating users to scan their spam folder regularly because spam filters will never be perfect. Then it will not be as big an issue if self-hosted email gets marked as spam, and the users of providers like Gmail can complain and push for filter improvements.

We have lost the old internet where you could have a first class host and interoperate with others. I just shared photos with my family and it was super easy… except my sister who uses Android. Everything is like this now. Big Tech is playing gate keeper and the days of rapid innovation are gone.

It would be possible to solve the spam problem once and for all with a crypto currency:

    if (Sender is whitelisted by receiver):
     All emails arrive in the inbox
    else:
     Sender has to send $1 for their email to arrive in the inbox
     The $1 will be returned if the receiver replies

Been self-hosting my email since 2018 on my homeserver. Luckily my ISP's static IP block is not blacklisted by any public list. Only had one instance so far where I couldn't register to a website because they were only allowing 10 or so domains from big providers in your address.

> The industry must self-establish clear rules which are harsh on spammers but give everybody a fair chance.

The author is dreaming, sadly. Why would the big email providers (corporations) change rules or do anything, when the failing system drives more people (such as the author) into their arms?

Funny seeing this on the top of HN when this is also on the top of HN: https://jan.wildeboer.net/2022/08/Email-0-The-Journey-2022/

Couple of suggestions: Oracle's OCI has a free offering. One could use that as a mail relay. Also SpamHero has been a pleasure to work with with very reasonable pricing to filter most of the junk that comes in, re-delivery service and outbound relay

How does one verify that their server has never ever sent spam, for instance through a security breach?

> You can no longer set up postfix to manage transactional emails for your business

Well, actually you can. But it's tough : https://news.ycombinator.com/item?id=20553028

Looks like the op decided to go w/ iCloud for the cfenollosa.com domain[1].

[1] https://files.littlebird.com.au/Shared-Image-2022-09-05-07-2...

Do not throw your towel.

I also manage since over 20 years my own mail server. Had a few problem but less than expected.

I love the flexibility and low costs besides the time which is needed to understand what is going. But this was a good invest if I look back.

Keep your mailservers running

I've hosted for several decades as well and intend to continue to exercise my right to do so. Whenever one of these monthly or bi-monthly doom and gloom woe is me posts shows up on HN; I wonder if there are any other motivations behind them.

I have self-hosted my personal email account for years without issues. At some point early on I started relying on AWS for sending emails, but that can be easily switched out or removed at any time (and its free for my small volumes of sent emails).

>You cannot set it up on your own datacenter.

There are still new email providers appearing every few years right? What special incantations are they performing to be allowed in the club? Do they buy large IP ranges? Do they pay protection money to Google et al?

As someone who was actually looking into utilising my own mail server for building my technical knowledge and understanding, I am dissuaded a bit that my efforts would end in a host of hinderances caused by some auto moderators and filters.

For those who are still self-hosting, what tools & services do you use and find useful?

I know the pain and the annoyances of hosting Email.

But still, I won't switch to big providers, I use Proton Mail personally, and Postale.io for many projects.

There is also mailbox.org and many others, you have a choice to not use the big providers, it totally possible.

“ This concept may sound familiar to you. It's called a racket.”

Wouldn’t RICO statutes apply then?

I use a third-party spam filter service, for both inbound and outbound. My MX records point to the service provider. I appreciate hiding behind them so to speak and I only open the firewall to the service provider.

Forget about self hosting. Even a custom domain is a pain sometimes.

According to a lot of web apps, my email is not valid and can’t use it.

Also I have been told by a customer support person that my email is not right as it has to end with gmail.com

My dad used to self-host email for his company. It worked fine for a long time -- until MS started silently discarding his emails.

A lot of his clients (very large companies) use MS for email, and his emails to them got silently discarded (not rejected as spam, not showing up in spam folder, simply discarded while responding that the email had been accepted). Notably, the invoices he was sending didn't reach the intended inbox, so he eventually had to move on to use another provider for email.

I've heard this story dozens (if not hundreds) of times over the last couple of decades. It gets worse and worse, since huge corporations only whitelist other huge corporations, and can choose to blacklist the rest. The oligopoly has won.

I can't send mail to mit.edu addresses. It gets rejected with a message that says I should talk to Microsoft.

I frequently end up in GMail's spam folders. No idea why.

I see no more than one piece of 'spam' a week; everything else is caught by a combination of a 15 minute greylist, the zen spamhaus BL, and SpamAssassin evaluating things. There are a bunch of spammers who send from accounts with valid SPF and DKIM, by the way.

Spam is a reasonably solved problem, but SPF and DKIM aren't much help.

(I get lots and lots of business spam, where the sender clearly believes that they have a right to try to sell me crap. The difference is that if I respond to business spam, someone will answer, trying to sell me crap.)

Sorry to hear you go!

I still self host my email. I ended up on a blacklist last year probably because some IP in my VPS block got marked. It sucked for about a week. But it came back eventually.

I run my own email server for receiving, but use the ISP's server for sending (and setting up Exim on Ubuntu provides such an option at installation time). I have no problem.

Despite my friends repeatedly marking my email as "not spam" google keeps classifying them as spam. Still hoping that one day European anti trust laws will fix that.

> There should be a recourse for legitimate servers

... one of the "big three" being google, this will never happen, there is no recourse in anything, even when you pay them for it.

The worst is Telekom in DE. They required you to host a website with your address, contact e-mail and phone number. If you don't, your e-mail is just blocked.

Everyone that can should host their own HTTP/1.1, DNS and SMTP at home on a battery backed Raspberry 1, 2 or Zero if they have an external IP.

Preferably with their own implementations, I have implemented all 3.

To host anything beyond those protocols and/or on more powerful hardware is often counter productive.

The problem is getting the ports opened, you need to fight for that right even if it makes spam worse in the short term.

Fight for external IP, ports and static IP in that order.

Edit: Reposting as separate comment because very important!

Edit2: How I wish downvoting required an argument. 500 karma is too low a barrier.

If some group wants to get serious about self-hosting email, first they need to define a new subset of the email standard for them all to conform to. Unencrypted C2S, unauthenticated senders, and outdated protocols like POP shouldn't be included. Make it simple to self-host.

XMPP has similar flaws except even worse. Too many different feature sets floating around, too many weird/insecure defaults. It also lost the federation game.

Think of HTTPS, a success story of self-hosting. The weird cases are just out of the question. You basically can't use encryption without signing, or old versions of TLS.

how many of posters here have used mailinabox ? or something similar? how was your experience?

i am almost 2 years into it now and beyond the first months hiccups, it just works

I was with the author up until the end. Why not have have regulation that makes the big company's spend money on processing? The can surely afford it.

How about each email recipient has an Absolute Whitelist and messages from addresses or domains on this list are always delivered come hell or high water?

I think the author of the article makes a few mistakes in his diagnostic.

You should never use a service where you cannot report a problem or where the administrators are out of reach. Of course it includes the "big techs" but they're not alone.

The underlying issue is that most of them delegate their filtering decisions to third parties. And many of them rely on the same centralized IP and domain blacklists.

Because of how the filtering is done, the End User generally never discovers that any filtering is happening. Only the sender may be notified by his mail relay of the delivery failure.

Of course, the blacklisters are not going to put the IP addresses of the big Email Vendors into their lists, if they did, millions of people would be notified of delivery failures with the risk of them discovering who is responsible.

""" Unfortunately, the computing power required to filter millions of emails per minute is huge. That's why the email industry has chosen a shortcut to reduce that cost. """

Even 20 years ago the computing power wasn't a problem. He probably has this impression because he's been using SpamAssassin. The real reason for why they are taking the "shortcut" is carelessness toward their service and users.

The excuse of saying that you should block messages before delivering them because it takes disk space is also heavily promoted by the blacklisters. Indeed, if the message was simply delivered to a Spam folder with the actual reason for which it has been classified as such, users could discover who is responsible for the filtering.

The good news is that there is some success in getting the big email services to remove centralized blacklists.

Another problem is that too many administrators of smaller services are not even aware of their reliance on blacklists. Sometimes this is because they have used an easy installation script for convenience, or because they've copy/pasted a configuration. And of course, there are those who do not understand the ethical implications of doing such a thing or are just foolish.

""" So, starting today, the MX records of my personal domain no longer point to the IP of my personal server. They now point to one of the Big Email Providers. """

This doesn't make any sense. The MX records are for inbound, not outbound, he could have used a different relay for sending mails yet still use his own relay for receiving (perhaps he doesn't know that?). Instead, he switches to a provider that is known for contributing to the problem. This is... disturbing.

G'Day Guys I have gone and had a another look at EFF and the issue for me is they have a 'Donate' in the menu. I'm a OG coder and have been giving away my efforts for decades, I believe that $$$ is the root of all evil, so I don't have any spare cash. What about a BC list of IPV4 that is a referral WL, I'm not gr8 at branding and don't believe in marketing, so the best I can come up with is 'BEN - Blockchain Email Net' ???

I pretty much stopped using email altogether once I realised this is what was happening. Maybe 10 years ago now.

Email was really useful if you wanted to send a message or notification to multiple people at the same time, but that got abused so much literally everyone disabled it, at which point email was no longer useful.

People can still send me emails (e.g. for plane tickets), but the chances of me replying to one are nearly zero. Now I send maybe 4 or 5 emails a year and only to people who wont use literally anything else.

Here’s an idea

Make a new protocol for this decade, that isn’t email.

HTTP is supported nearly everywhere SMTP is. Just build something over that, and this time around make sure to avoid SPAM bullshit.

People shouldn’t be able to just message you based on your address. They receive a capability to email you. People can be empowered to give out your capabilities. If a particular such branch leads to spam, you simply cut off that branch and boom, no new user can reach you with that capability anymore. Hashes of Public keys can identify users.

There are companies that claim to offer decentralization by hosting them on their own servers. I think we've already lost.

Alright blockchain nerds, now is your time to shine. An actual problem that can be attacked through distributed consensus.

I threw in the towel a few years ago, after 15 years running my own server on top of dan bernstein's qmail.

We need kubernetes level of distributed messaging system to beat email. What’s most popular next option?

Is email itself broken? What could replace it, that isn't a "platform" like WhatsApp?

Email is broken. Fix that first so that hosting it isn’t such a terrible chore.

Why can't we have an open source mail hosting solution that self-updates?

Who is any middle man to decide for me what is spam? Send in the regulators!

How to charge people for sending me an email? With hashcash or something?

Hey [0] built a very successful email business in only a couple of years with no deliverability issues. There are hundreds of thousands of happy Hey users who continue to pay for email. So much for anticompetitiveness. There may be a difference between a guy with a server and a real business serving thousands of customers.

[0] hey.com

we shold come up with a more universal approach to tackle spam

seems to me that email transport is by now a fairly robust and universal delivery mechanism at least just above the IP layer, and it's utility decline creates the perfect opportunity to piggyback a new system on top of the existing infrastructure:

high concept: encrypted email with cryptographic postage stamps paid directly to the recipient.

Want me to read your email, pay the postage, and the same vice versa. Who cares what gmail is doing, I don't want them collecting my postage anyway.

this would simultaneously bootstrap a universal micropayment currency

Seems like we need to write a law (draft a bill).

We need new protocols. Blockchain protocols can somewhat solve the spam problem by charging to send. Receiving addresses can also whitelist or charge to receive.

Why are we still using email?

No, I'm serious, why?

Another compromise between self hosting and using "big tech" email providers is to use smaller email providers. I have been using https://kolabnow.com/ for a five years or so and am a happy customer. Of course it's more expensive, has less bells and whistles, etc, but I'm glad I don't have to participate in the surveillance capitalism machine of big email providers.

how about trying a distributed censorship resistant version like skiff.com?

Sad, but true...

This was a great read. Thanks for posting it here!

I found this line to be especially intriguing:

> Hellbanning everybody except for other big email providers is lazy and conveniently dishonest. It uses spam as a scapegoat to nerf deliverability and stifle competition.

The big tech firms criticized in this article are guilty of these sorts of transgressions in other arenas, as well. It's always been my contention that the "hellbanning" of user-generated content by big media and big tech alike comes from the same motivations. YouTube and CBS alike want to make niche content difficult to consume in order to stifle any competition that might get vaulted up as a result of that niche audience finding the new distribution endpoints. This comes with the added bonus of reducing cost of goods sold, by reducing the firehose of new content to process. Or, as the article puts it:

> Unfortunately, the computing power required to filter millions of emails per minute is huge. That's why the email industry has chosen a shortcut to reduce that cost. The shortcut is to avoid processing some email altogether. Selected email does not either get bounced nor go to spam. That would need processing, which costs money.

I would be very curious to learn if there are any proposed explanation as to why this phenomenon is so commonly spread throughout the big tech space. Do we get the same kind of behavior out of other enormous multi-national firms like oil producers, ocean freight companies, defense contractors, and chemical suppliers?

bashblog!

Totally agree with the tone & premise, but

> At some point your IP range is bound to be banned, either by one asshole IP neighbor sending spam, *one of your users being pwned*,

feels like a hint that

> My current email server IP has been managed by me and used exclusively for personal email with zero spam, zero, for the last ten years.

Might not be entirely accurate.