Everything was set up properly. The domains I hosted had IPv4 and IPv6 addresses with forward and reverse DNS matching exactly. SPF was set perfectly, according to multiple test sites. DKIM was always used and always validated, again according to multiple test sites. I even signed up for a DMARC endpoint account at report-uri.com.
Didn't matter. Messages would be spontaneously rejected, sometimes in mid-conversation (this happened to my wife quite a lot). Other times, the remote end would claim to accept the message and then it would disappear into the great bit bucket in the sky. It would be like this for a day or two then spontaneously go back to working.
I finally gave up and switched entirely over to Fastmail awhile back. No more delivery problems, but no more self-hosting my own e-mail, something I'd done for twenty years prior (my domain is older than Mozilla and Google).
Email was actually one of the first aspects of the "classic" Internet to fall to the gentrification of the tech monopolies, long before the web in general. In theory, it's an open ecosystem anyone can jump into. In reality, unless you have a lot of energy/time/money to get around the big providers banning you for "spam," it's not going to happen.
To be fair, I was also surprised to read that anyone was still using Hotmail, so maybe I just don't have my finger on the pulse of the hip new generation.
Not for me and loads of my tiny and not so tiny customers. Just keep your nose clean and follow the rules. This is in the UK, elsewhere ymmv.
Well I've been self hosting my emails for 10 years now, and my emails still go through. I'm sure there must be cases where you face issues, but not my in my personal experience.
Only thing I've seen once or twice are websites with "validation" rules like /.@(gmail|hotmail|yahoo)./
Was like 1997. Needless to say it was then sendmail struggling to get the mails over dialup. At least my single-threaded Netscape was responsive.
Worked until 1999 or so. Then spam became a thing and more and more mails were rejected and I folded my setup.
A couple years ago I did some superficial analysis on the domains of the registration emails our customers were giving us. I don't remember the details, but IIRC only about 80% of them were connected to one of the big providers (google, apple, qq, microsoft, yahoo(!), etc).
But isn't the federated nature of email the only thing which prevents the said monopolies from taking over email completely? If it can be proven that X giant is blocking mails from Y small email company then they're in lot of trouble at least in EU.
That's the reason chat and everything else is a lost cause; Can't send a message from Messenger to Telegram unless you use their bot API to build a specialized application[1] which is a downgrade from XMPP like interoperable protocol.
[1] A product from my previous startup which allowed communication between Messenger <-> Telegam <-> Viber <-> LINE for dating - https://hitstartup.com/about/#FindDate
The problem is that big players (Gmail/MS) care about an anti-spam false positives only when they affects a relatively large fraction of all messages. Small mail servers account for a small fraction of all messages, so big players don't care about them at all.
I will not be surprised if they have SLI/KPI which allows them reject by a mistake millions of non-spam messages daily because it is a small percent of all messages they process.
For some time seemingly legit mail from Azure Cloud Services also got flagged by Gmail. If that was the case -- I thought -- what changes do I have for my own hosted or non-major email provider? That is when I decided I couldnt win as a small startup and just went the managed GSuite route for email hosting.
Apparently the service as a whole has been based on o365 since around 2015 according to wikipedia.
Iirc at its height 90% of all email was spam.
Can anyone speak to what a provider like Fastmail does behind the scenes that guarantees their delivery? Is it the threat to sue/kick up dirt/raise issues with the appropriate regulatory body if perfectly good emails don't go through? Merely having a seat at some sort of regulatory body or committee?
I don't know what other measure they may take but this alone probably goes a long way.
And yes, this is fucking frustrating. I'm not going to send thousands of mails per months just for the sake of it and don't know how I should do that without spamming. This is so fucked up.
All opinions are my own and not that of Microsoft.
The only part of the article that surprised me is the fact they actually got the issue escalated. I've never succeeded in doing that; talking to Microsoft support is like talking to a brick wall.
I once had DKIM keys get out of sync with DNS in Office 365. It’s supposed to be handled automatically and that should never happen. Support was as useful as a baby trying to do calculus. It took me 3 days to get the keys rotated and after I understood how it works I realized they should have been able to help me mitigate it right away (dkim keys can be rotated while disabled).
I’ve found that the best trick with Microsoft is: money talks. If you pay them for support suddenly you should have a much easier time getting things done.
Obviously paying MS to fix a problem they caused is abhorrent but sometimes you gotta hold your nose and do it.
My organisation used to self host everything -- wonderful times; everything worked, but with a HTML 1.0 crappy web interface. About four years ago they shifted to Exchange (urgh -- suddenly IMAP errors and some of my message-read state information became lost. I also noted the massive decrease in my Mac's battery life as Mail seemed to have a lot more work to do -- but I digress) and about two years ago self-hosted exchange became O365.
They recently started pushing harder into moving everything into Microsoft's cloud, and my quality of life has massively decreased -- the first time I was driven insane by it was when my colleague's human-written emails to me ended up in my spam folder; and vice versa when I emailed six undergraduate students to set a date and time for a tutorial. I turned up, they didn't, as I'd been shadowbanned from sending a five-line message to recipients within the same organisation -- messaging them all together got it dropped, sending it individually without the CC was fine. I complained and got given the run-around for a few weeks with no answer. The response of my local "IT people" was "not our fault mate".
The other thing I particularly hate about Microsoft 365 is that all the user-facing error messages are "Sorry, something went wrong". I'm not an idiot, please don't treat me like one.
Having to write this is so ridiculous. Better just use a throwaway and be yourself. Or is this PR?
PR will mention a minor technical problem that has been solved, nothing to worry about.
Not sure how things are organised in microsoft, but it's one of worst support ever happened to me
Couldn't a matrix client with email and asynchronous features be a drop-in replacement for email ? Wouldn't that be a killer product ?
SPF, DKIM, and DMARC are not to core to email protocols if I understand correctly.
There shouldn't be a way for a matrix a client to delete a message sent to another client to keep with the email usage and expectation. So matrix messages wouldn't necessarily be used as email text messages but attached documents to messages would carry email that a client would store and archive.
As the article mentions, Microsoft’s server was telling the sender that it had accepted it. An ACK had taken place. But there’s no point in having an ACK if the recipient lies about it.
Most probably this whole problem realted to silently dropping emails is intended behavior, a type of shadowbanning applied to emails to prevent spammers from knowing their email was blocked.
Too bad that this in turn causes huge issues for legitimate users
In general, I think whatever you are suggesting will not have all the same guarantees of delivery as email.
If that were true there wouldn't be silent drops in the last mile.
> In general, I think whatever you are suggesting will not have all the same guarantees of delivery as email.
ACK is not to guarantee that messages are delivered though. It's a mechanism that ultimately would let the sender know the message was most likely not delivered.
Let me know when that new client is default installed on the phones and laptops of, say, 70% of all people. I’ll start taking a look at it and evaluate it against standard smtp/imap/pop3 then.
But in these days and age of webmail I am not sure 70% of people have a mail client.
I don't really understand the argument of pitching mass adoption against something that doesn't exist yet.
Unlike virtually all other email providers, Microsoft does not cooperate if you follow standard email practices. You can follow all their instructions, sign up for their monitoring tools, submit issue reports, try to build "reputation" and if you are self-hosting they will still have a high likelihood of silently dropping your emails after their server tells you it got them, or putting them in the recipients spam even after being marked as an approved sender.
Just ignore that it’s Microsoft breaking a legacy standard to advantage their other products that compete against email. I think they should start fining companies in stock. Dilute the shareholder value by 5% and see how quickly things change.
See also (recent): https://answers.microsoft.com/en-us/outlook_com/forum/all/ho...
Imagine if this was turned around, and simultaneously a huge chunk of the world instantly black-holed Office 365 outbound email.
Then in response to the roaring noise from Redmond, simply send back their canned response of "Not qualified for mitigation. Our investigation has determined that the above IP(s) do not qualify for mitigation."
Do that for 24 hours, then 48 hours, and so on until they smarten up.
I guarantee you they'll have their CEO breathing down the necks of every moron in the spam filter team until they fix it.
Until then? Nothing will happen. Nothing at all. We'll complain on YC News, we'll complain on /r/sysadmin, and a bunch of small businesses will go out of business. Microsoft 365 will continue to grow to consume the world. Email will be rebranded "Teams Mail", and as far as everyone is concerned that will make perfect sense.
Either we act, or we live with the consequences of inaction.
"As far as I know, emails from my business custmers are branded, formatted following best practices, have unsubscribe links and a valid contact footer."
In other words, his customers are sending out the type of emails that actually require unsubscribe links. You know, _spam_. So maybe it isn't Microsoft who is the bad guy here?
> Despite all the outrage
Hah! Please, there's hardly any. Certainly nowhere close the outrage this deserves.
So yes, telling people that their spam-load was accepted, and then just not delivering it, seems perfectly fine to me. It's a tough world out there, and it was made that way by precisely the kind of people that need "unsubscribe" links in their emails to begin with. I feel no sympathy for him.
These days communication uses web forms, Teams, Skype, Facebook, Twitter, or any other electronic mechanism that isn't email. They are all regulated by their respective owners, so they have a slightly better chance of surviving the onslaught of the advertisers. Email works fine for contact with friends and family, but not for spam, and that's how it should be.
I use Outlook and the few companies who still pull this shit off end up in the spam box courtesy of machine learning. If MS is making life worse for businesses they have my support.
Translation: "The filtering is based on machine learning. If there are false positives, too bad, even we don't know what's causing them. Try to look less like spam or something?"
They had a dedicated page at https://sendersupport.olc.protection.outlook.com/snds/data.a... that supposed to be used for these delivery issue. I tried to submit multiple requests from there.
Even IPs that aren't sent any email in last 3 months.
Then one day the problem just disappears and I was able to send email just fine. Till this day, I still don't know what happen.
Another similar service is apple icloud, which I think even much more worse than Hotmail. At least, hotmail give you a portal that you can see some data.
With the announcement of apple icloud email service, I imagine people who are self-hosted email gonna have a lot issue with them.
What announcement?
iCloud has had a E-Mail service for over 10 years now. iCloud was doing email back when it was called MobileMe.
So that means one more awkward email providers we have to deal with :)
The reason for blocking IP originates from a culture set before DKIM. Any IP could shoot out emails for any domain. You don't want to blacklist a domain for an email it did not legitimately author.
However, in a post DKIM world, where adoption is above 80% this could be a feasible solution at least for emails carrying a DKIM signature. The domain owner has provided proofs that the IP address is being used by them to send out emails. Flagging emails from a domain and IP could be blocked while allowing other domains from the same IP to operate.
You'd be surprised (or not) by the difference verbal communication makes, not only do your verbal expressions translate better, but the tone of voice may (or not) instill trust in the person on the other end.
It's working just fine for most people. Email sometimes isn't working for Microsoft customers - but that is entirely Microsoft's doing.
Adding a relay user to the o365 then configuring all the domains i needed to send from on the o365 tenant (without changing the MX records naturally). (Using a normal o365 user account to relay is possible for me as our typical customer base is enterprise and education, not random end users) Adding distribution lists as the from-addresses that i needed to send from and gave send-as rights to the relay user, compiled a list of hotmail/outlook domains (which i've had to add to over the years but it is relatively stable for me by now), then configured the outgoing relay server that does all the dkim-stamping to route all outgoing mail to outlook/hotmail domains through o365.
When i see a S3150 in the logs i just add another domain to the list, i should probably make this pre-emptive by looking at the target mx records before sending, as its not trivial to make exim resend something that ended in a 5xx.
All other mail providers have removed blocks by simply asking, not hotmail. Also all our newsletter-like mail goes through separate services, these are all transactional mails (meeting invites, user account creation, password resets and such).
I think I have it configured now to alert the recipient and me on quarantined emails but there are at least two relevant dashboard (Exchange and umm Security Center? who knows what the current names are today) that probably have like 35 submenus and 1000+ settings. Running my own email server would be easier.
Another thing, a few months ago I noticed a couple incoming emails were not quarantined/blocked but also not delivered. I don't think the issue has reoccurred since but it's just crazy that emails are just lost sometimes.
Was told my email server was sending spam by them (as people had reported emails from the IP as unwanted), so I did a full security check (using logs, trying to find holes, etc), and upgraded software. No problems that I could find.
After lots of back and forth, they said my IP was blocked as it was "suspicious" and they unblocked it (how kind of them). The process took a long time, which I could imagine could be devastating to a small business (luckily my client understood as I made sure to tell them how bad Microsoft is).
Silently dropping mail (not even to the spam folder) is a maddeningly cryptic action on MS's part, and has squandered massive amounts of developer and system administrator time.
If you're in a position to sway anybody from using Outlook.com, Hotmail.com, or Msn.com or Live.com I highly encourage it - MS has made a lot of lives difficult with their email policies, including NerdAdmin's here.
Also, seen this happen with other apps too. Transactional messages don't arrive on Microsoft email addresses.
Contacting Microsoft means you have to go through a difficult dance through multiple canned replies, multiple escalation requests, and being lucky enough to get someone to actually escalate.
Silently dropping emails isn't Microsoft-specific, though. It's quite common to have a Postfix/Amavis setup set to discard spam over a certain spam score threshold.
We have been sending login code emails via AWS SES and have had a lot of issues with Hotmail/Outlook.com not delivering them.
My takeaways: 1) There's a risk that a shared IP system like AWS SES will get blocked by SmartScreen based on emails someone else is sending. 2) We really should look at the contents of our (trivial) transactional emails, too.
It's been enough of a headache that I haven't been able to migrate off GMail and that's frustrating as hell. Especially with important medical emails..
I'm coming from the perspective of a community site owner that pushes out email notifications to its members. Nothing spammy, all emails are opted-in and essential for the basic functioning of the community. They may be as simple as "person X liked your post". It's purely functional email. Plain text, no links, images, nothing. The entire site is non-commercial.
Further, the domain from which it is sent is well established (14 years old) and never used for sending out spam. It also has implemented all the best practices, like DMARC, SPF, the like.
It's not in the gray zone by any stretch of the imagination. It's as clean as it gets. Yet still, as random as the weather, it's always hotmail.com and live.com users complaining about email not arriving. It's not in their spam folder, it's just not there at all.
I never face this issue with any other email provider, only Microsoft.
There's no help at all regarding what triggers it, other than the standard document listing best practices I'm already following. They do have some tools to analyze issues, but guess what, only available for high volume email pushers.
There's also a public tool to check blocklists, some of which are shared between email providers, yet even when not blocked anywhere, you're still blocked at Microsoft.
So after all this self-analysis, I've learned to raise the ticket. Admittedly, occasionally they admit it was a false positive, and then lift the ban. But they never fix anything structurally, it's an issue that keeps coming back. It's maddening.
Since I'm in the mood of self-pity, I'll add to this Microsoft-specific problem by mentioning that it's hell to push out email in general for small websites.
For example, besides the functional email notifications, there's also an opt-in weekly newsletter. It's found deeply within user's personal settings, so those finding this setting and enabling it, surely want it.
Within the range of a few hundred to a few thousand subscribers, now you have a signal problem. A handful of users may no longer want the newsletter and do not unsubscribe (which is easily done from the bottom of each email). Instead, they may delete it, report it as spam, block the sender.
These are convenient to do from an email client, but the problem is that I never see their desire to unsubscribe. I'd gladly honor it. What happens instead is that in the case of low volume, just a handful of these negative signals may push all of your email into the spam folder.
Technically this is incorrect. It's not spam. The user opted in, the contents of the emails are not spammy, and the user never indicated to not want them anymore. But email providers don't care, it's all spam now.
To resolve this, I compare last activity on the site with the user's opt-in status, and auto unsubscribe everybody not recently active, regardless of whether they want this. This significantly reduces the chance of users flagging the email as unwanted in their email client. At the same time, now I have users complaining their newsletters aren't coming.
Bottom line: you can't win. With the best of intentions, fully wanting to comply with every technical and legal guidance in the book, it still does not work.
Misconfigured email servers, with spam filters and SPF/DKIM setups being the most common issues, drop emails far more often people tend to realize.
