JMAP is, from what I’ve read, a great protocol for building an E-Mail (and now also others) client on top of.
Since I would like an innovative way to access my E-Mails, but do not want to self-host, I would find it interesting to use Stalwart as the server component of an E-Mail client: Data is somehow synced into Stalwart via the “ugly” protocols and I get a nice API to build an elegant client on top of.
My basic research shows that something like IMAP-IMAP sync seems to be a thing. Has anyone done something like this, perhaps even with Stalwart? (this of course grows in complexity for each new protocol to be proxied).
I believe having this kind of setup easily accessible could jumpstart a new generation of E-Mail clients on top of JMAP because it (relatively elegantly?) circumvents the chicken-egg problem by allowing all existing IMAP mailboxes to be accessed via JMAP.
I also did not realize it is… primarily developed by a single person? The author. Incredible:
https://github.com/stalwartlabs/stalwart/graphs/contributors
I was originally thinking you'd need to go remote IMAP <-> maildir <-> Stalwart IMAP, which would be really complicated, but I think the IMAP <-> IMAP should work fine.
With Stalwart in place, there’s finally a reason to develop a client for JMAP.
I hope y’all are aware that Mozilla’s new mail service will use it, so that is likely going to give JMAP a big push!
Nextcloud was such a terrible experience for me (the file sharing/storage was good, but the groupware aspect was incredibly buggy). But knowing that Nextcloud is partnering with Stalwart to hopefully overhaul their stack, Opencloud is developing their JMAP integration, and Mozilla/Thunderbird is using it too (they already have a webmail in development here: https://github.com/thunderbird/stormbox)... we might finally see some exciting development in this space. And now is also a ripe time, as there seems to be a perfect storm of people wanting to get away from Big Tech platforms.
Just to clarify. Stalwart is the first to have JMAP contacts and calendars. Cyrus has had JMAP mail since the beginning of JMAP.
It's the first implementation of JMAP applied to stuff other than Mail.
You can now synchosise any combination of CalDAV, JMAP, and filesystem.
Still, it’s an interesting space, I think.
Email was never a binary protocol. Notoriously so, it's why MIME types and MIME encodings get so complicated.
Most of the "old internet" protocols (email, FTP, even HTTP itself) were bootstrapped on top of built-mostly-for-plaintext Telnet. HTTP as the new telnet has a bunch of improvements when it comes to binary data, request/response-based data flows, and some other considerations. HTTP/3 is even inherently a binary protocol, it's lack of "telnet-compatibility" one of the concerns about switching the majority of the web to it.
vCard/vCal/iCard/iCal were also deeply "plaintext formats". JSON is an improvement because it is more structured, even more efficient, than those predecessors. JSON may not look efficient, but it compresses extremely well and can be quite efficient in gzip and Brotli streams.
I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet", even if it doesn't sound like "binary protocol efficiency" at first glance. Especially as HTTP/3 pushes HTTP more efficient and more "binary", and arguably "more fundamental/basic" with HTTP/3 even taking over more roles in the TCP/UDP layer of the internet stack. (Telnet would never try to replace TCP.) HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of. Sure, it would be neat to see more variety and experiments outside of the HTTP stack, too, but HTTP is too useful at this point not to build a bunch of things on top of it instead of as their own from-scratch protocol.
I made up ULFI because I thought MIME has some problems.
> JSON may not look efficient
Efficiency is not the only issue; there is also the consideration of e.g. what data types you want to use. JSON does not have a proper integer type, does not have a proper binary data type (you must encode it as hex or base64 instead), and is limited about what character sets can be used.
(Also, like other text formats, escaping will be needed.)
> I feel like "JSON over HTTP" is a subtle improvement over "custom text formats over telnet"
I think it can be, depending on the specific use; sometimes it isn't, and will make it worse. (HTTP does have the advantage of having URLs and virtual hosting, although I think it adds too much complexity more than should be needed.) However, I still think that DER is generally better than JSON.
> HTTP isn't the worst bootstrap layer the internet could use to build new protocols and apps on top of.
I think it depends on the specific application. However, even then, I think there are better ways than using HTTP with the complexity that it involves, most of which should not be necessary (even though a few parts are helpful, such as virtual hosting).
That's just not true. Telnet and SMTP are built on top of TCP. They live on the same layer. They were originally both protocols that transmitted data with printable ascii, hence why they look similar. There are many other protocols like Telnet and SMTP that worked like that, auch as nntp, irc, and yes, even http.
It shouldn't. For some cases it helps, but other times it doesn't. Sometimes it helps but there would be better ways to do it, making it on a simpler protocol or making an entirely new protocol (which might or might not use TCP; sometimes it is better to use TCP and sometimes not) depending on the specific case.
> Stuff like file sharing or groupware, mail, calendars, and so on—these things could be a lot more efficient and don’t really need the overhead of JSON as the message interchange format, IMHO
I dislike JSON. I think it has many problems, and that DER is a better format.
(There are also the "small web" protocols such as Gemini and Scorpion and Spartan and Titan, which avoids some of the complexity of HTTP; I had considered using DER-over-Scorpion rather than JSON-over-HTTP. It is also possible to use SSH, although SSH does not have virtual hosting.)
I'm struggling to think of any real benefits to not using HTTP other than it would be more interesting.
HTTP/2 and HTTP/3 are binary protocols. And if you replace the JSON with CBOR, then even the payload becomes binary.
The reason for using HTTP is that the semantics are right. HTTP is a state transfer protocol, and ultimately, that's 90% of what you need for sync.
The other 10% is for subscriptions, updates, with versioning, and patches. You can get these by adding the Braid extensions (see braid.org) which upgrade HTTP from a state transfer to a state synchronization protocol. (I work on Braid.)
Absolutely yes, IMO. This significantly eases web client development.
JSON by comparison has simple, obvious limitations that more people are familiar with dealing with.
There's also the tendency to tie your protocol to implementation. The Microsoft Exchange "protocol" didn't get reverse engineered for so long because it's basically the COM structure of Outlook fed through (if I remember rightly) DCOM-RPC.
There's no magic. Nothing sacred. Nothing that you aren't allowed to understand, intuitively. Nothing where you aren't allowed to imagine "what if it also had X?" The web is yours. The computer is yours. As an industry, we burn some incremental percentage of bandwidth to give you the keys to the kingdom, and to allow you, new developer, to be one of us.
In an age when LLMs feel like magic boxes to tech-minded people new to development, we need this more than ever.
https://datatracker.ietf.org/doc/draft-ietf-jmap-calendars/
And Contacts was only 10-months ago.
The downsides of developing and testing this stuff as we were writing it up!
We've finished rewriting the objectid generation to give smaller sized and more sortable IDs (they're an inverse of nanosecond internaldates now, plus some extra magic on the low bits for IMAP appends to avoid clashes)... which we wanted to speed up and reduce disk usage for the offline mode.
Next up is indeed updating to the latest spec on calendars and contacts. Files might take a bit longer, I really want to do some desktop clients for the files system, we have a really nice files backend at Fastmail which is only accessible via our interface or WebDAV right now.
The next, next big thing would be the Chatmail relays[1] supporting JMAP based servers (right now it's Dovecot) and this new targeted push extension for faster notifications without battery drain on mobile. I can see how the Fastmail mobile client will benefit from this RFC as well (it's already incredibly battery efficient, thanks to the team).
I looked into adding JMAP support to Thunderbird but the client is so tied around the ideas and principles of IMAP, it needs surgical refactoring of many parts of it and I don’t love C++.
So instead in my spare time I am developing a JMAP only gnome email client, using many Stalwart libraries. Think Geary but Rust instead of Vala, GTK4 instead of GTK3 and JMAP instead of IMAP. It’s been mostly an excuse to play with Rust and gtk-rs and Relm4 (beautiful Elm inspired rust bindings for GTK4). Someday, it will be released.
Client support for a new protocol is never that quick, but I believe adoption will happen, at least outside of the big providers, who will never support it.
There is no good desktop implementation of MUA with old technologies (IMAP, Sieve), will all this JMAP help?
I don't think so.
What is profit to have good server with new good (assume it is good, I'm not sure, but lets assume) protocols without good client?
IMAP4 is underused by modern clients: it allows to effectively store client configuration on server, nobody implements it on client side. It allows to configure per-folder Sieve scripts, nobody implements it on client side. Nobody implements good Sieve client (with folder name autocomplition and such) even for global script, not to mention per-folder ones. Heck, there is no good Sieve editor! (I know about Sieve client built on Electron, it is not good, it is incomplete and buggy).
Servers are solved problem (sendmail, exim, postfix, dovecot, cyrus). Clients are not, they stagnated at the moment GMail was announced.
Clients on the other hand have actually kinda moved forward, Apple Mail works with IMAP servers and offers features that people only got with Gmail before. But there are many other examples as well.
I don't have any problems to use UTF-8 in my messages received via dovecot and I have some folders with national characters on my account, and it works both with IMAP and Sieve.
You need both. You could say, what profit is a good client without a server? By that reasoning, we never stake a step forward without a complete solution.
Now a better mail implementation is just a client away.
Maybe, Apple Mail is good, again, I don't know as I'm not using Apple, but I don't know any client for Windows and/or OSS with Windows support which has as basic features as support of per-folder settings (identity, sorting mode, etc.) stored on server or proper support for Sieve. In my eyes it is basic features.
And don't let me start to rant about message editor itself, especially in text-only (as opposite to HTML) mode with proper quoting & such.
P.S. ("New" Outlook already only connects to MS365 servers and then stores your credentials and data on Azure, while they proxy to your actual IMAP/SMTP server )
(This should not be interpreted as a defense of IMAP.)
You don't need major providers to support it, they support SMTP and that's how messages are relayed. JMAP is just so you: the client, can fetch your mail from wherever you host your mail.
For regular desktop software, I’m not sure that it’s really an improvement over existing protocols.
I’ve got a friend who’s been pitching me on building a new email client for years. “I’ll do it if we exclusively use JMAP.” “okay does that include Gmail and Apple/iCloud accounts?” “Nope.”
I could sort of see dual-supporting Gmail's proprietary API and JMAP, but unless the #2-5 competitors support it… what’s the point? (sorry to put on the pessimism hat)
To be honest, I’m not sure why end-users would want JMAP for e-mail access.
It would be interesting if they do successfully roll out all of these additional RFC proposals providing a cohesive “groupware” protocol covering calendering, contacts, file shares, etc, we see notable server implementations, and interest is enough to drive client support.
That’s a lot of “ifs”.
The documentation is not great - I'd say it's just about barely enough to get an overall idea, but there's no one proper single definitive overview of what options exist, what are their possible values, what are the defaults, and how they relate to each other. Maddy docs, despite looking a bit sloppy, were a lot easier to get through. IMHO Stalwart makes it unnecessarily difficult to write a non-minimal static configuration file, hooking everything up correctly.
To be fair, maybe there is a page like that but I haven't found it, despite trying.
I know the Web UI allows to do the configuration by clicking through the forms, but this approach conflicts with declarative deployment practices. In my case it's giving me nondescript 500 errors in the UI with "Failed to write local configuration" in the logs because the .toml file is read-only.
But in general, I agree that it has not been a very smooth experience. Having messed around with maddy and mox, Stalwart has had quite a few gotchas. Despite being a single binary promising simplicity, I'm finding it to be a real challenge figuring out how it all fits together, and I'm mostly learning by trial and error since the documentation is often outdated.
My biggest gripe is that it doesn't use the config.toml for every setting, or at least doesn't seem to have the option to do so. I broke my installation and had to find the posgresql key-value pairs for the settings, which was made harder by the fact that everything was stored as binary, which also made me have to edit it as binary as well. These were very simple settings that would have been a breeze in a flat configuration file. I absolutely do not like how necessary the WebAdmin is to manage simple things.
That said, the integration with calendar/contacts is nice even without JMAP... Getting Thunderbird and Roundcube setup with plugins and proper settings made it so easy to get several users setup with calendars, contacts, and shared email-boxes and shared contacts right upon first login.
The S3 storage is also working great (Hetzner Frankfurt VPS paired with AWS eu-central-1), and AWS downtime a few days ago notwithstanding, I'm feeling good about the reliability that gives me, leaving me mainly with the PosgresQL data store the main thing to keep backed up.
This is a hugely ambitious software and as such, there will be many things that I will have a hard time getting used to as a hobbyist, but also a lot to be gained. I'm sticking around for now and waiting for version 1, improved documentation, and more clarity on how it all works.
I have asked sooo many times since Stalwart first was introduced, but not got a straight answer. It is just FastMail or Topicbox. I want something like roudcoube or wildduck that can be used over https that I can self-host!
[0] https://github.com/cypht-org/cypht
[1] https://jmap.io/software.htmlFor example, it automatically handles Let's Encrypt certs for you. You get JMAP, CalDAV, WebDAV, CardDAV, IMAP4rev2, DKIM/SPF/DMARC, MTA-STS, DANE, spam filtering, SQL+blob+object storage backends, search, clustering, OpenTelemetry, etc all in one tiny binary.
Downsides: some features are gated behind an enterprise version and I think the dev team is one guy, or at least it was a while ago.
Having ran both for a long time, I'm sticking with Stalwart from now on as long as development continues.
I haven’t been there in more than a decade. I really am curious what the response in Apple (and Google) is to this spec.
I researched what it would take to implement a full calendaring server once, and after reading all the RFCs, just backed away slowly from the whole idea and never thought about it again.
Never hosted Postfix / Dovecot stack, in fact this is the first time I host emails, but from what I understand Stalwart is designed to handle inbound directly.
For very high throughput inbound you could check out KumaMTA - it was designed specifically for that, but I think Stalwart doesn’t have bottlenecks in it’s clustered topologies which would require it unless you are doing something crazy.
They have very good docs in general IMO, here are docs on how to cluster - https://stalw.art/docs/cluster/configuration
Haven't looked into spam more closely yet. After first glance on most publicly shared email address - there is around 2 spam messages per hour.
Here is report prepared by llm which looked through the last 20 email headers found in spam. All of them were categorized correctly, however there were few emails in the past few days which went to spam where they shouldn't but I think this is fixable.
- Critical Authentication Failures: A large number of the messages failed basic email authentication. We see many instances of SPF_FAIL and VIOLATED_DIRECT_SPF, meaning the sending IP address was not authorized to send emails for that domain. This is a major red flag for spoofing.
- Poor Sender IP Reputation: Many senders were listed on well-known Real-time Blackhole Lists (RBLs). Rules like RBL_SPAMCOP, RBL_MAILSPIKE_VERYBAD, and RBL_VIRUSFREE_BOTNET indicate the sending IPs are known sources of spam or are part of botnets.
- Suspicious Content and Links: The spam filter identified content patterns statistically similar to known spam (BAYES_SPAM) and found links to malicious websites (ABUSE_SURBL, PHISHING).
- Fundamental Technical Misconfigurations: Many sending servers had no Reverse DNS (RDNS_NONE), a common trait of compromised machines used for spam.
There have been few messages which went to spam which didn't meet any of this spam criteria but actually they were cold marketing emails, so it's good too. In addition to this stalwart emits info log for each possible spam message ingested. Not sure if this can get any better than this.
Nylas pricing has gotten better recently, but is still quite high though - at $1.50/connected account/month at scale, it's likely material to your per-user margin if it's part of your SaaS offering.
But if you have a use case where this is a no-brainer (like capturing/analyzing/building custom real-time UI around your internal sales team's emails) then it's remarkably powerful.
edit: we use it on very resource constrained environments, the container version is too much overhead.
This is not the case for all versions, but I've found it to be common enough that I have to read all of the release notes between point versions when upgrading.
It can definitely be improved.
I am most curious
That said, I'd give this another shot. What I discovered is that Stalwart is incredibly flexible and designed to be used in any configuration you want, and the documentation and examples are incredible, compared to most other projects. BTW, there is no webmail yet, but you are right that there is the admin that is served over HTTP/S, as well as MTA_STS, JMAP. The default listeners get you started and it's kind of out-of-the-scope of Stalwart in terms how you want your server to interact with those listeners. But whether you containerize it or setup a reverse proxy, the documentation has a lot of examples, including how to pick up the certificates, if not managed by Stalwart itself.
I landed on a Caddy for HTTPS and haproxy for proxy protocol on just JMAP, and the remaining TCP services directly binded to the host for the main mail services.
Traefik is another example that I had initially prepped and got working that also handled TCP connections with proxy protocol where needed. The Stalwart documentation was very helpful with that.
In short, I was impressed with how easy it was to integrate Stalwart into whatever setup I wanted, and how open the developer is to different setups. For instance, mox (while awesome in its own right) considers containerized setups to be not-recommended, and generally expects it to live on its own server. Stalwart is flexible, but the side effect is that it is overwhelming at first... but becomes quite elegant once you get the hang of it.
1. curl this shell script
2. Run it as root.
Anytime you see that, you can assume the software wants to take over the whole box and isn't likely to make any attempt to play nicely with other services/software/users already running on the same host.
To their credit, they do have a docker image (https://stalw.art/docs/install/platform/docker).
Can others confirm if these problems are widespread? I get that these protocols are probably a pain to develop for but given they are "robust, widely adopted and battle-tested" it seems that is probably a solved problem. It's better to have one standard that is used everywhere than to have to choose between two standards.
Always relevant: https://xkcd.com/927/
I guess contacts/calendar follows JMAP naturally when the clients already implement it, but that only applies in the 'already wrote a JMAP email client' case. Virtually any other case would rather stay with widely supported protocols?
However, doesn't stalwart already also support WebDAV though?
I think we're about ten years past the point where "newer = better" was a reasonable starting presumption.
JMAP is better than IMAP because IMAP is a too stateful design, the IMAP/SMTP distinction allows for misconfigurations where sending doesn't work, has dozens of extensions where key extensions are inconsistently supported, doesn't have as many batched operations, etc. One could make an effort to improve IMAP - but the effort to do this consistently in server software would likely be comparable to adding JMAP and the result worse...
OTOH, the new protocols intrude on areas that go far beyond email software (you're very unlikely to get support for these in older Androids/iOS/Windows even if the modern OSs ever consider them), and don't offer as much as JMAP offers over IMAP. The cost/benefit is worse. They may make sense for a JMAP email client but IMHO not elsewhere.
Also, after code the CalDAV ↔ JSCalendar part, using only IANA time zones instead of scattered ones in CalDAV components makes things much simpler.
> Stalwart Enterprise leverages AI technology to provide unparalleled email security and management. With AI-powered features, Stalwart Enterprise excels in accurately classifying spam, detecting sophisticated phishing attempts, and blocking various types of network attacks. This intelligent approach ensures that your email environment remains secure and reliable. Stalwart Enterprise comes equipped with a pre-trained large language model (LLM), offering robust out-of-the-box protection. Additionally, it supports integration with leading AI providers such as OpenAI, Anthropic, and other cutting-edge platforms, allowing you to enhance and customize your security measures. By utilizing AI, Stalwart Enterprise delivers a smarter, more efficient email solution that proactively safeguards your communications and data.
It sounds awesome but the way it is intro'd here:
Over the past few years, the IETF has been redefining how email, calendars, and contacts are synchronized and shared. Building upon the success of JMAP for Mail, several new protocol extensions have been introduced:
JMAP for Calendars - A modern replacement for CalDAV and CalDAV Scheduling.
JMAP for Contacts – A powerful alternative to CardDAV.
JMAP for File Storage – A replacement for WebDAV-based file storage.
JMAP Sharing – A modern successor to WebDAV ACL.
JSCalendar - A clean, JSON-based evolution of iCalendar.
JSContact – A modernized, JSON-native successor to vCard.
Sounds more like the parts of the web dev that give me ick (new and shiny; rush to copy new and shiny in other contexts; give it a year; and all of a sudden only 1 of the 6 actually was successful)
The modernization efforts of JMAP are interesting, too. Most of the old protocols are a mess of bespoke plaintext formats full of quirks evolved over decades in a giant mess of different software. Even the stuff that was already web tech like WebDAV and its extensions CalDAV and CardDAV were full of quirks, violated some REST "rules", and originally intended for a different purpose (file shares/FTP replacement). JMAP is much closer to "plain REST" than WebDAV's complex HTTP protocol extensions/changes.
https://datatracker.ietf.org/wg/jmap/history/
Bron is the principal of fastmail, who now own pobox. This is a serious activity.
That's a really cruel response, because this is important work. I don't want my kids beholden to bigco.
I think it's real & important.
I also wanna make sure people like me, who have to keep tabs on the intersection of "how can I help liberate from BigCo" and "how can I make a livable wage doing so"
It is, quite literally, real, but also something you shouldn't waste time on if you're already busy. (c.f. https://jmap.io/software.html)
Now JMAP is quite a bit nicer to use than IMAP's API, but IMAP's gravitational field is too strong to be supplanted. IMAP is also becoming somewhat of a niche protocol, as the majority of users use vendor proprietary protocols for accessing their emails on Gmail, Outlook/Hotmail, etc. So why invest the time to add a niche replacement for IMAP when the entire protocol is a second class citizen to mainstream email clients.
If you want to push a new technology, you need to start somewhere. That's exactly what's happening with JMAP. It was created by Fastmail to use as a bridge between their servers and their own apps a case for which popularity doesn't matter. It's basically a modern vendor proprietary protocol but done in the open.
From there, support is only a matter of someone being interested enough to implement it and manifestly it's working. There are now three servers (Apache James, Cyrus and Stalwart) and some clients.