Major European payment processor can't send email to Google Workspace users (opens in new tab)

I once had a job where reading standards documents was my bread and butter.

SHOULD is not a requirement. It is a recommendation. For requirements they use SHALL.

My team was writing code that was safety related. Bad bugs could mean lives lost. We happily ignored a lot of SHOULDs and were open about it. We did it not because we had a good reason, but because it was convenient. We never justified it. Before our code could be released, everything was audited by a 3rd party auditor.

It's totally fine to ignore SHOULD.

for the client. If you're implementing a server, "the client SHOULD but didn't" isn't a valid excuse to reject a client either.

You can do it anyway, you might even have good reasons for it, but then you sure don't get to point at the RFC and call the client broken.

Sure, you can argue, if you require that the email reach their destination, it is required to set this. ;-)

But I am totally with the OP here. SHOULD was never a requirement, just a recommendation that is maybe better to follow.

But that means a valid reason could exist and Google would block those mails too.

Must = external requirement

I cannot fathom how you think should* would act as a requirement in any sense of the world.

Anyway, in general you can expect that doing unusual but technically valid things with email headers will very often get your messages rejected or filtered as spam.

For consumers, ignoring a SHOULD mostly affects their own robustness.

But here Google seems to understand it as a MUST... maybe the scale of spam is enough to justify it. Users are stuck between two parties that expect the other to behave.

So at that point the ID has no value to me except being obliged to carry it around with the message, so maybe the originating system can at some point make sense of it. But then there is obviously no reason to ever reject mail without it, it's an ID valid for the sender and the sender didn't care to include one, great, we save on storage.

Should in most RFCs also mean "do it as long as you don't have a very good technical reason not to do it". Like it's most times a "weak must". And in that case the only reason it isn't must is for backward compatibility with older mail system not used for sending automated mails.

And it is documented if you read any larger mail providers docs about "what to do that my automated mails don't get misclassified as spam". And spam rejection is a whole additional non-standardized layer on top of RFCs anyone working with mail should be aware of. In any decades old non centralized communication system without ever green standards having other "industry standard/de-factor" but not "standardized" requirements is pretty normal btw.

Mail RFCs do not cover at all spam detection and malicious mail rejection, but it's a thing every large mail provider has and you really have to care about when producing automated mails all looking similar. And large mail providers like google tend to document what "base line" of additional requirements they have for accepting (automated mail). Having a Message-Id is in there, and in pretty much any larger mail providers documentation about that topic. Tbh. I have worked with mail before a bunch of years ago and the need for Message-Id was back then really no hidden gotcha but pretty well known.

and the design space mail provides is larger then any client could reasonable support (like it's really a huge mess covering docent of standards which allow all kind of nonsense and hypothetical use cases practical unsupported), so you anyway have to look at "what everyone does" and only then make sure it's also RFC compatible, instead of starting with the RFC. That was a painful lessen to learn.

In addition there are some de-facto standards not pinned down in any RFC, like e.g.:

- Message-Id being required for any automated mails by many mail providers (through how bad the consequences are if you don't have it diverges largely).

- You can't punycode encode the local part of an email address (it would be a different email), and there is no standard way (as far as I remember) to convert non us-ascii local parts to us-ascii. This is based on the fact that iff your mail server allows you to have non us-ascii local parts it should also support "internationalized mail" (SMTPUTF8 and co.). But it's a semi industry standard to give the user with an unicode local part also the mail with the punycode encoding of the local part so it often "just works" and dev are frequently surprised when it fails to work...

- You can have quoted text in local part, like whitespaces. But most of the industry decided to not give users such mail addresses so you can see it as soft deprecated and using it is asking for trouble.

- Attachements. The MIME encoding allows a lot of different ways to put mails together and doesn't force a specific semantic interpretation by mail clients. As such you if you naively use it you might run into surprises how/if your attachments or embedding(s) are displayed. Through today embedded resources often are either not done or uses data URLs. Again which ways work well and which don't is somewhat an industry standard and not in any RFC.

- A lot of different ways to encode Unicode to us-ascii. If you produce any mails you probably should by default use the latest revision (where encoding is often just not needed as things are utf8), but might need a fallback with it often being fully unclear if . But if you are a client you probably have to support older versions. And in some parts of the world/business segments usage of very very old mail servers is a thing which is a major pain if you run into it.

so quoting that something isn't strictly required by mail RFCs is kinda pointless as even many things explicitly allowed won't work well in practice

As a rule of thump: If you can afford it test you system will all widely used mail providers as if you where an external customers. And redo the tests yearly.

oh and as a bonus, if you mails looks too similar to known phishing mails it will also just disappear. That seems irrelevant, but e.g. if you use Keycloak with the default mail templates for password reset and co. there is a high chance of your mails ending up in spam or not even being delivered as scammers have used Keycloak for their means, too. And that isn't just a case for Keycloak but any "decently widely used open source software producing mails and having default templates". So you pretty much always need to change the default templates (you will do so anyway for branding, but skipping it in the earliest stages of a startup where branding might still be in flux isn't that uncommon either).

And we should be raising hell for it. Should never happen. Using your popularity to violate protocol should be not be tolerated

Once you deviate a bit from the standard, you're down a slippery slope. Its not that difficult to use pragmatism to justify wrongdoing.

Some people have become way too comfortable taking for granted that it is okay to treat others in an uncivil fashion. To those I say: keep it up, and we shall revert to the fundamentals of society where violence is an option and one day you are copping punches in the face, or getting shackled and thrown in a ditch somewhere for the ravens. Behave like an animal, get treated like one.

You're precisely right that customer experience matter, but I wouldn't put it past some conservative European company to go: Well Google is wrong, so they should fix that. Google doesn't care, you can't make them care, you can't even contact them. Just make it work for your customer.

Google and Microsoft's email practices define a pseudo-RFC in practice. As an engineer, I hate this. As a civic participant I can vote against it. But as a person that sells my software services for a living, I am going to implement the Google/Microsoft standards to the letter, not argue about definitions in an RFC.

Another example may be a lightweight implementation of a spec in a limited and/or narrow environment, which remains technically compliant with full implementations of a spec but interaction with such a limited/narrow environment comes with awareness about such limitations.

But they just did (make it work). The logical assumption is that most ppl did the same, just used another email provider. Why would viva care? (same as google, why would google care?).

Or you could pass it through ~5,000 lines of C [1] and you will have it done in milliseconds even on hardware that would be old enough to drink.

[1]: https://codemadness.org/webdump.html

- You are sending a receipt and want table alignment for items

- You want to put a logo of your company so that readers can recognize who's the email from

- You want to make unsubscribe link smaller and the "open the thing I'm notifying you about" link bigger, so that people would know which one is which without reading the url

- You want to add a header

This is about media types, not encryption.

It's amazing that society even functions at all.

Incompetence and corruption only slightly overlap in most cases, i.e., being competent at corruption is a very real thing. The incompetently corrupt, usually end up punished... and there are few and far between...as we all very well know.

The kind of schemes you mentioned are generally not going to be how "corruption" will manifest itself in European financial institutions, because although it is also difficult to speak in general across Europe since the EU has not yet subsumed democratic self-determination all across the continent yet, so there is wide variation; the competent corruption is largely in the form of money laundering and tax evasion, not lower level quantitative schemes that would quickly come to light because Europeans are also a lot more cognizant of money and value than Americans, so people are paying attention a lot more closely and will raise hell over a single cent, where Americans are known to have hundreds of dollars draining out of their pockets every month just alone on recurring payments for things they don't even use anymore and don't bother dealing with it.

What we all don't really seem to internalize as a human species, is the absolutely demonic type of pernicious nature of "banking", i.e., a kind of LotR, ring, that consumes you especially if you are weak... and human, or at least European civilization seems to frequently go through periods of immense weakness where things are going springily and everyone is dancing to the music the "bankers" are playing as they are pandering our pockets, and when they realize they could get away with that and all the pockets are plundered, they move on to plundering our homes, then our accounts, then they want to take our first born... "Banking" is like humanity's cocaine, the seemingly innocuous, feel good drug that will consume your soul if you do not rage and fight against that demon taking over aggressively. It's no coincidence that cocaine is so widely used by the most parasitic elements of European societies, especially in "banking"/finance in general.

it differs by country (EU gives countries some similar overlapping laws, but they still are distinct countries with a lot of different practices and different laws in many many ways.)

and it differs by severity

E.g. deutsche bank has become notorious to appear in most large scale bank scandals _in the US_ as a "German Representative". They also repeatedly get into trouble for relations to money laundering operations and other issues. At the same time they are the Bank which has to give you a (expensive but not completely absurd priced) bank account no matter your credit score, criminal history etc. and is the bank you mostly likely have a escrow account parking money if the person who gets it "if you mess up" is the state/local government (like e.g. in some long term visa context).

In general banks move very slow, including with software protocol updates/changes. And that isn't just the case in the EU AFIK. Actually some of the most crazy cases of "old software" I have seen in "the west" where as far as I remember in the UK and the issue was that it was deeply fundamentally impossible for them handle a lot of non English names correctly, like in 2017~2020 or so. Through at least they still could use the bank, just with a crippled/en-ified name. Funnily the same but worse (as in not working at all) I have heard about Japan banks, where they had(have?) really big issues if you have a middle name.).

lol... how much time do you have? Here's just one to start with off the top of my head: https://en.wikipedia.org/wiki/HSBC#Controversies

doesn't sound like a hard requirement

But hey, if you're in a business domain where categorically leaving 6 million potential clients-who-are-demonstrated-to-spend-on-things isn't an issue? One fewer thing to worry about, right? ;)

Comments on issues use the format <[OrgName]/[RepoName]/issues/[IssueNumber]/[CommentID]@github.com>

A mitigation to this would be to take the combination of message ID and the sending domain and use that as the unique value, because message ID is not guaranteed to actually contain a domain label that's owned by the sender.

For example SendGrid's message IDs are <[RandomValue]@geopod-ismtpd-[Integer]>.

Microsoft regularly sends legitimate emails from Microsoft to my spam folder.

Even with a six-figure email spend and weeks of troubleshooting the best response we could get from our mail provider was that they were having problems getting traction with Microsoft on the issue.

Message-Id being required for automated mails is a de-facto industry standard

while the consequences differ between mail provider, it missing will also make it much more likely for mail to be reject or put into the spam folder

It's also well known. Pretty much viva engineers fucked up doing proper research.

Now to be fair:

- it sucks that you can't just implement the RFC(s)

- the standards suck, docent of different RFCs overlapping and replacing each other and referencing often older versions of other RFCs, with docents of ways to do the same things of which only some can be used reliable in practice and a common gaps in the standards about edge cases or about the "higher level semantics" of constructs.

- so overall mail seems very simple at first but if you want to automated send mails reliable internationally it's a total pain and Message-Id is just the head of the iceberg.

so if you (ab-)use protocols for mail clients to send automated mails they might very well not add Message-Id. In general many mail providers have both a way to send a mail where "they do some clever parts" (like adding Message-Id) and interfaces where they mostly just send what you give them. It's possible that they migrated from one solution which did automatically add Message-Id to one using interfaces where it doesn't happen without realizing that there is a mismatch in this solutions do implicitly add.

In my case, it was reportedly (for MS) an IP associated with mine (same hosting provider) had previously been used to send spam.

My domain is decades old, never sent any spam, and I whitelisted it .. but nope, my host wasn't perfect.

This was some time ago now, but it looks like they've still not adopted proper whitelisting.