It started as an experiment, asking myself; can we store "state updates of a datastore" in Matrix instead of chat messages? Now, I'm convinced it's actually a really powerful combination to develop real-time, collaborative software.
I'm using it for a new project and so far didn't have to write a backend yet. Matrix takes care of a lot of stuff: Authentication, E2EE, federation, hosting, etc. - so I can focus on the client.
I love the ideas of Local First [3] software. Personally, I'm convinced the technologies in this ecosystem (CRDTs, etc) are really powerful and can do more to decentralize software than many web3 technologies that currently receive much of the hype.
However, it's still early days, so I decided to open source parts that I think will be useful to others (I recently also shared SyncedStore [4])
[1] https://github.com/yjs/yjs
[2] https://www.matrix.org
[3] https://www.inkandswitch.com/local-first/
[4] https://www.syncedstore.org
I get that Element needs to focus on one use case in order to pay the bills, but there's a lot of room for applications that have barely been explored - calendars, contacts, games, social media, heck - you could build a federated gitlab-like collaboration site with truly federated data. You get multi-device sync, e2e encryption, and identity management out of the box.
Thanks for putting this together, I hope there will be a gold rush as people realize how much low-hanging fruit there is!
I'm interested in this just to keep my devices sync'd up. I know Samsung/Apple/MS etc all have syncing services, but I really really want to get all my data out of their ecosystems.
Just a distributed calendar app would be pretty amazing. No more iCal subscription/export/whatevers, 'just' federate the calendar between devices, and arbitrarily share it with anyone else.
I have no clue where to start with building an app on these abstractions. Does anyone have any pointers? Documentation to read? High level overviews? Etc.?
Matrix-CRDT posted here should abstract most of the nitty gritty stuff, but there are no shortcuts to designing with CRDTs so familiarizing yourself with those is important.
Other than that, just start playing and asking questions! The #matrix:matrix.org room is full of friendly and helpful people and I'm sure you'd be able to get answers.
Like Matrix, JMAP is commonly misunderstood, with people thinking it’s about email (it’s replacing IMAP, which is Internet Mail Access Protocol, right?), but it’s actually fundamentally an object synchronisation protocol, with the core in RFC 8620 being domain-neutral, and the email model completely separate in RFC 8621. (IMAP has also shifted a bit over time from being about mail with proper synchronisation being an unsound and secondary concern, to nailing down sound object synchronisation in extensions. But JMAP does it better.)
Matrix is about decentralisation while JMAP is client-server, but you can probably reuse much or all of the underlying data models, just as JMAP for Calendars and JMAP for Contacts are establishing JMAP-independent data models models first.
- What happens with E2EE when there are key exchange problems? (i.e. a server goes down and the account is moved)
- What happens when someone redacts a message? I assume you'd want to disable redactions for these rooms or you'll end up with broken update trees!
- How do you communicate clearly that any data you add is in the event chain forever? In Word and GDocs you can remove old revisions of a document, but I don't think you can in this system? That could be a feature, of course!
- What's the performance impact of such a system on a server editing reasonably large documents? If twenty people each edit ten to twenty documents you end up with quite a large set of state, especially over time!
- If you'd enable E2EE for the Matrix client that you pass to Matrix-CRDT, key management is covered in the same way that Matrix does it normally. If key sharing with a particular user is broken, then you won't see updates from that user anymore and v.v. Basically, the transport between you and that user is broken. As CRDTs such as Yjs are designed specifically to work Local-first, at the core there is no assumption that all clients should always be connected to each other. Once the clients are able to communicate again, potential conflicts would be "resolved" according to the CRDT design.
- Redacting: yes, basically you need to trust clients not to fiddle with messages (I think this is fair as you trust them to work with you on the same data already)
- UX / communication: good question! Technically would be possible to purge old (deleted data), but I think this is still something we need to explore together while we start to see more mature software built on these technologies.
- Kevin already answered your performance questions. Matrix-CRDT makes an additional optimization so that not the entire history of the Matrix room needs to be retrieved (see "Snapshots" in the readme)
Additionally, exposing the CRDT stream would allow for reactive index building for searches on the data (ie. timely dataflow operations to aggregate an index).
Relm [2] even models a 3d world using Yjs. I don't necessarily recommend doing this as 3d applications usually produce a lot more actions per minute than text applications. This required some workarounds and deep knowledge of how Yjs' optimizations work. But it's definitely possible.
[1]: https://blog.kevinjahns.de/are-crdts-suitable-for-shared-edi... [2]: https://www.relm.us/
I have said it before and I will say it again, I think this year is going to be the year of Yjs. The ecosystem around it is brilliant and I believe it will become the leading CRDT framework (it probably already is).
Happy to see you using TipTap too for the demo, a brilliant rich text editor framework (with support for Vue.js, React and Svelte) built on top ProseMirror with first class Yjs support.
The TipTap guys (Philipp and Hans) along with Kevin Jahns of yJS have stated the y-collective inorder to centralise funding and organising of the Yjs ecosystem:
https://opencollective.com/y-collective
Personally I'm excited about the us of Yjs and CRDTs outside of just rich text editing, I think there is great potential to build a distributed offline enabled database targeting mobile and PWAs with it. Something like PouchDB but with automatic conflict resolution.
Finally there is a Rust port in development to improve the (already very good) performance and make it cross platform with other languages.
BTW, we are already tinkering on some interesting stuff with the Rust port, too. :-)
Take a look at https://concordant.io
Instead, I want to encourage you to build an API that others can use to efficiently store shared data. Feel free to ping me if you need input.
- Kevin (Author of Yjs)
However, not all CRDTs are the same (actually, I think the definition is still somewhat vague - most things "eventually consistent" could be called a CRDT). Yjs is specifically designed for high-performance operations and has great support for rich-text collaboration. It also works great while offline - and you can connect different providers as you like (for example, store updates both locally in IndexedDB, and transmit messages over websocket / webrtc / and now, also Matrix). Definitely wanted to keep this flexibility.
That said, using this to stream CRDTs, and THEN federating those streams between servers would be pretty wild.
It’s called TypeCell as that’s the (yet to be announced) project both Matrix-CRDT and SyncedStore are building blocks for.
If you’re interested in tech like this and maybe also “future of programming” (those bret victor things ;) ), knowledge management software (notion, roam, obsidian) - i’m sure you’ll like it!
When you auth with Matrix, is that a centralized action (one company / org behind it) or is that a decentralized action (its more a protocol for auth amongst the community of users)?
ie. Is using Matrix for auth decentralized or a different centralization?
This design is fairly similar to email where you can be @gmail, @hotmail, or @whateveryoulike.com
100% agree! Web3 is really only tackling monetary instruments, which is interesting but very narrow. The majority of apps people need don't require global consensus, and it's too slow and expensive anyway. Yet, there's so much interesting stuff that can be decentralized/federated.
> Matrix takes care of a lot of stuff: Authentication, E2EE, federation, hosting, etc. - so I can focus on the client.
This is great to hear! What else besides CRDTs do you think is needed in order for matrix to be a platform for general purpose apps like this?
Some nice things:
- Anyone can host their own Matrix homeserver, which can then sync its data within a federation, allowing users pretty fine-grained control over what data they share and with whom.
- Add end-to-end encryption, and it becomes even better: data stored somewhere in the cloud (i.e. the user doesn't have to manage a server), but the server knows almost nothing about what you've stored aside from some metadata.
I have a prototype running that uses Matrix-CRDT and Matrix E2EE, and it worked great. It's still a bit of a hassle to set up though (mainly, configuring the matrix sdk correctly), I hope to make that easier later this year.
I'd probably have to re-architect the whole front- and back-end though. Although on the other hand, there will be some code editors (currently using Monaco) where this could be usable.
https://github.com/replikativ/datahike
Is an Open Source variant of Datomic.
Lambdaforge wants to eventually have this work with CRDTs.
Using the Matrix ecosystem for this is quite interesting as it solves many problems for you already.
I've thought about datomic + crdts in the past, and it seems like ":db/unique :db.unique/identity" properties would be an issue. Whether or not the transaction fails depends on the current content of the database.
I don't know if there is a way around this (while still being a CRDT) or how necessary this feature is.
1. It would be easy to manage data sharing. Post updates to data in channels that the relevant people have access to.
2. It would be easy to debug data changes. I could see it becoming the equivalent of a terminal app but for the web.
[1] https://en.wikipedia.org/wiki/Conversational_user_interface
It works beautifully with almost no code. I used Yjs P2P, but P2P is blocked in numerous networks.
If you want to try, with an hackernews room, but sorry it's in french : https://nosgestesclimat.fr/conférence/hackernews
So using Matrix as a backend is a great idea.
The only caveat I found to Yjs is that if you want to persist data (and not lose it when the server crashes), nothing's really plug-and-play, so I went with Supabase.
