Lessons learned from creating a real-time collaborative rich-text editor (opens in new tab)

I guess you have to make your whole frontend open source if you use their editor under that license? I'm not sure how GPL interacts with scripting languages. Do you have to run GPL scripts on a GPL'd interpreter for it to be permissible?

Regardless, the pricing for their commercial offering seems really intense. $29/mo for 25 MAU? I feel like they're missing a few zeroes there. Maybe they meant 2,500 MAU? 25,000 MAU? The demo I played with makes it clear that the editor component is rather nice, but I would rather look at alternatives at that price. I would imagine the collaborative nature is what influences the price, but somehow Google Docs and MS Office do fine with collaboration as a core product feature without charging >$1/MAU just for that feature.

Here's an explanation: https://www.gnu.org/licenses/copyleft.en.html

Oh, that's really tempting :) Believe us! There are people in our team who would love to dive deeper into CRDT. We confirm that OT definitely has its problems. It's elegant at first but gets really dirty the deeper you dive into this... hole. Especially the undo feature adds to it a tone of complexity.

So, some of my colleagues see CRDT as a potential saviour. There was even a paper on CRDT for tree structures that we might use (can't find a link now). However, we'd rather go with a linear model for CRDT. That'd make it a bit easier on the CRDT level but it would require some really smart decisions about: types of changes, the structure of the linear model and how it is transformed to the tree model (that we need after all). We know, though, that without an actual implementation, it's just a theoretical discussion. Therefore, we started thinking whether we could add CRDT "below" our current tree model. That might allow validating the concept in a real-life. Can't wait to do that :)

BTW, in the meantime, we recommend this paper: https://arxiv.org/abs/1810.02137

For anyone interested in CRDT, I can recommend the deep-dive writeup to Raph Levien's Xi editor:

https://xi-editor.github.io/xi-editor/docs.html

Ditto. Would love to hear more about the OT vs CRDT decision!

As a gentle introduction to OTs, one of my colleagues did a Rails Conf 2018 presentation on implementing collaborative editing. His blog is here https://blog.aha.io/text-editor/ including a simple implementation in Ruby.

Differential synchronisation works very well indeed for plain text, but as Neil himself has pointed out, it doesn't necessarily work well for tree-structured data (like XML or JSON) because there's no guarantee that the combination of separate syntactic-validity-preserving edits will necessarily be syntactically valid.

Nice! It's my understanding that the default method used by Neil requires a fixed polling interval, and thus updates are limited to every 100ms or so. Did your implementation get around that?

Hi!

Certainly, there are different use cases and different expectations. What we call "offline collaboration" can be seen as a special case of the real-time collaboration. In "offline collaboration" you also need to merge changes so you also need the conflict resolution. However, the longer you postpone merging, the better the conflict resolution algorithms need to be because the more conflicts you have. That's one of the reasons why we introduced new, semantic operations.

The other option is so-called change tracking or suggestion mode. That's on our roadmap as well and thanks to the platform we created it's not a big deal now. In fact, if we started with implementing suggestion mode (or "offline collaboration"), we might have ended up with an engine which is not ready for real-time collaboration. But once we have real-time collaboration we can now be quite certain about solving other issues.

if jira and confluence allowed this my company would benefit from probably a double digit efficiency boost.

we have to split our "collaborating" to google docs, and our "carving in stone" in JIRA and Confluence.

I'll say though that the "suggested edit" and "comment on a range" features (and the distinct edit modes about commenting, suggesting and approving edits, and free for all battle royale are all valid modes for us all the time.) (300 person SF+intl. software company)

I showed the demo to a passing PM and asked him how long he would expect a team of five developers to spend on it and he said "a couple of weeks... no wait, maybe two months." :)

show this stats next time the junior dev will say that she/he will develop own WYSIWYG editor in like 1-2 weeks for the project needs ;)

That's an interesting opinion that differs from my item. My impression to all software costs, not just this one, is, "Wow, software is just so damn cheap!"

Compared to any physical project or product software is by comparison so cheap!

It truly is the most interesting subject I've been working on yet. The fascinating thing about it is that it isn't a "solved" issue with one, correct solution, so you can feel like a 19th-century scientist at times :).

If I were Google I'd buy up CKEditor to help compete with Microsoft Office.

Among the many other problems with your proposal, you are assuming an editor that can only insert and delete characters, so no selection, copy, paste, search and replace, and any other features that involve modifying more than just one character in a document at a time, since you're critically depending not just on the fact that all edits can theoretically be represented by single character changes but that the rate of such changes must necessarily be very, very low, bounded by human typing speed.

"OK, so to fix that, I'll just..."

That'll break in some other case. If you then work on fixing that up, in another 20 or 30 steps, you end up at these algorithms. Or something far worse, which is actually pretty likely if you try to bodge something together one hyper-local problem at a time.

Ok, so what happens when you insert a character?

That sounds like a horrible UX

Interesting question! Actually, at the moment we don’t :).

We did some testing to see how big this issue is and it turned out that it's not dramatic. The main difficulty with graveyard purging is that you need to keep the undo in mind. Elements in the graveyard are used for undo, so garbage collecting will have to be strictly connected with limiting undo steps.

Add to that collaborative changes and it becomes a tougher problem. Therefore, we'd probably start from what seems to be the safest option – trimming the history stack and implementing a garbage collector that goes through all the possible reference points.

It seems that the link is fine. Scroll a little down and you can test the collaboration in Letters (built using CKE5) or with CKE5 document build (switch the tab).

Thanks! A good UX was indeed one of our main goals and we weren't sure whether we're not overdoing this (again, the Netscape history ;|). But we kept in mind that simpler solutions already exist and it doesn't make sense to just copy them. That'd make the job easier, but we'd end up with the same issues that we identified in the existing platforms (annoying conflict resolution, stability issues, platform limitations to linear content – no tables, no nested blocks). So we risked taking the long path. The UX served as a compass and allowed making some reasonable shortcuts in some non-critical places. If we'd try to solve all possible issues that we've hit, we'd still be in the starting point :D

Our first idea was to have four basic operations (insert, move, attribute, remove) and then have “deltas” (wrap delta, unwrap delta, merge delta, split delta) which were built using those operations and for which we described additional transformation cases. In this approach operation transformations were set in stone, but you could add your own deltas and provide your own transformations for them.

In the end, we concluded that users will see defining deltas and transformations as too complicated and won’t use it. Later, we also decided to drop deltas idea at all and we rewrote deltas into operations.

As an alternative solution, we provide the post-fixing mechanism. Your plugin can listen to all the changes done on the model and apply fixes if something has gone wrong. We used it for the famous "insert table row / insert table column" problem, to add the missing cell. This alternative is not as clean as explicitly providing a transformation algorithm but it is much easier to implement.

The problem with defining your own operations is that you need to write transformations against all the other operations which is a huuuuge work. It took as a few years and we still see some room for improvement.

While I generally agree they are nice, one nit is that some of the fonts are unnecessarily small and/or faded. It looks to me they could be made more visible without sacrificing the visual utility of the diagrams. Roughly 1/3 of the population would have age-related vision difficulties with those kinds of fonts. Every graphic designer should take courses in accessibility.

Recently we realised that our public communication about collaboration is quite confusing ;( We mention "Cloud Services" everywhere, but the reality is that you can have an onpremise installation of our collaboration backend that you can host/control by yourself. The SAAS offer was first, that's why the current website looks how it looks. The onpremise offer is currently in a closed private-beta state, so if there is anyone willing to give it a try, we're open. We'll make it fully public in a month or two, after we polish the documentation and so on.

I would think yes. Structured editing is tough because of the interaction possibilities. Think of it in simplistic terms "add words to paragraph" at the same time as "remove paragraph." What do you do?

With text presentation and reading, you are ultimately limiting the number of things that can be seen that way. Though, you can of course expand your entity list to include more and more things. Headers, lists, sections, chapters, paragraphs, etc.

That is to say, I think this is already difficult for structured text. Adding more structure, in the forms of "requirements section" and such just compounds on that complication. I'm not sure it adds more.

The typical way I've seen this used is almost like pair programming. One user will write a draft of something, or add an image or chart while on a conference call with the other editor(s). Honestly, I see it a lot more with spreadsheets than with rich text, but the use case is there. It was one of the features that put Google docs on the map back in the day.

As far as I remember shareJS was available when we started to work on tree-based OT. We researched it but it turned out to be too simple for what we wanted to achieve and there was no special handling for some edge cases that happens during collaborative editing.

We achieved something similar to shareJS functionality quite early, to be honest. What proved to be extremely difficult was all the extras to smooth up the user experience. We would have to (heavily) extend shareJS anyway, so it made more sense to go out with own solution, crafted for our needs.

Graveyard is just another root, like the main one. So, the same way that all elements have the same position in the main root for all clients, elements in graveyard also have the same position for all clients. Thus, their path is kind of UUID for them.

Hey Tade0! As much as we love CKEditor 5 that we created, we also love our users who are still widely using CKEditor 4. Some of them invested many, many months to create custom plugins, adopt CKEditor to their systems and so on. So, being a responsible company, we will maintain this product still for several years. As a company that creates components, which are embed into other systems, we have to simply deliver stuff that others, including big enterprise customers, can rely on. I know it's a bit unusual in JavaScript world to maintain software for 8(!) or even more years, but this is what we do. Yes, we're a bit crazy.