Honestly I don't understand anything. I am not able to make sense of the data flow and the architecture. I want to understand how text editors work under the hood. Also I want to understand the plugable architecture they use to extend the functionalities of the editor.
Please suggest me any articles, videos, conferences, blogs, where I can pick up the concepts. I have been troubled by this lack of knowledge and unclear path to access it.
Edit: Reasons for this quest: I am not here to create yet another text editor. But I do understand that they are one of the complex peice of software which still is under constant improvisation and developement. Also text processing is the one of the core concepts of computer science. A lot of algorithm and data structure knowledge is hidden inside it. Besides, I feel through real world projects one can learn alot about core computer science foundations.
I suggest this as it will force you to solve problems yourself as opposed reading about other people’s solutions. Basically learn by doing. Learn by struggling to come up with solutions and data structures, thereby developing some insight. The stepwise development should help focus on a small subset of problems at a time. Don’t be afraid of changing data structures as you gain knowledge. In fact write code to make it easy! Use a language that won’t trip you up in low level issues such as memory management. The api will make testing easier. Armed with this knowledge you’ll better appreciate and understand other people’s solutions as well!
memmove() is about 100 gigabytes a second on my computer. That's 100 megabytes every single msec. My screen refreshes at about 100hz. None of my source code files are even 1 megabyte let alone 100 megabytes, so at least on my computer, for all of the files I edit, there isn't a single operation that a "rope" or a "buffered view" could offer over the naive because I could literally just rewrite the entire file in C 100 times every keystroke, and I think for many operations it is obvious how to do better.
For this reason and others I think it's absolutely plausible for a beginner to self-teach themselves how to write a good text editor and even improve on things, and I would never want to discourage someone from a discovery they have both the interest and the time for.
The process of incremental building things. Figure out what needs to be built first and then implement it. Finally compare your work with others. To be frank I used to feel that there must be some sort of deep intellectual concepts lying around, and whatever I try or implement is just dumb. But now after what you described the process and technique seems not so complicated.
https://github.com/antirez/kilo
Even I could understand this one pretty well and that's no small matter.
Less does almost everything an "editor" does, at least visually, except change text. It pages through text, forward and backward, line by line, it handles lines that are too long, tab expansion, it searches, even has an extended command set. (Can less do syntax coloring?)
It also handles files too big for memory. These are all editor problems. Mind its solutions may not be optimized for an editor, but it's certainly smaller.
Today, modern machines "suffer" from "too much" performance which actually frees you from not having to worry so much about the actual backing store, especially early on. Do you really intend to be editing a 2GB file? Honestly, how big is an average text file? And how many billion cycles per second does a modern CPU handle? Sucking the entire file in to RAM, and just pushing stuff around with block moves will take you very far on a modern machine. Not that you should not look at the other data structures (there are many), but you don't have to start there, depending on where your interest lies.
Also consider hunting down the book "Software Tools". There's two editions, the original and "Software Tools in Pascal". It's by Kernighan and Plauger. They go through in detail and write a version of the 'ed' line editor.
And if you really want to work on an editor, the CP/M world would love a new one. There, it's all about efficiency.
I don't know about most people, but I deal with such files (usually JSON) on a weekly basis. It's surely one of the things that makes implementing text editors tricky.
I, too, have dealt with huge files. My primary large file use case for vim is as an interactive grep, opening files, then just performing iterative filters reducing the file to something "manageable". If the file is too big, I might resort to raw streaming filters to temp files before going all vim on it (even vim has issues with huge files -- huge files are issues all their own).
But it's more a matter of pragmatism as to how far one wants to take a pet project like this. There's a curve of diminishing return depending on what someone wants from such a thing.
A single RAM buffer is a bad idea, to be sure, but raw horsepower of modern systems and CPUs cover up a lot of bad ideas. Simply the idea that you COULD mmap or even malloc a 2G buffer to suck the file in, and it would WORK, is enough to make ones teeth itch. I remember long a ago an article from someone encountering an early workstation with 1G of RAM, and the worlds it opened up. We have come far from those days.
If the OP is interested in buffer mechanics, then starting with an API backed by a simple buffer goes a long way. If they're interested in screen painting, window management, etc, then the RAM buffer may be all they need. Otherwise, they can work on replacing their back end with all of the assorted structures folks have mentioned, see what they like best, as they all have tradeoffs.
And I regulary work with JSON files the size of some MB, which already puts some editors into a real struggle.
Out of curiosity, why do you have so big files? (A whole DB as a json file?) And what editor do you use for them?
Has anyone used other mechanisms for large document types? It seems like an array of paragraphs, each containing a simple string could probably handle most editing tasks and might be easier to layout and manage. Curious what other data structures people have used!
The other big reason to prefer a rope is that the worst case complexity is excellent. Basically all incremental operations are O(log n). With an "array of paragraphs" you get various pathological performance cases such as a huge number of small paragraphs or one very big one.
A good rope implementation is not trivial, but when done right it hides its internal complexity from the layer above. And it's a solved problem. There are at least two or three solid rope crates for Rust (just to pick the language I'm most familiar with), and will likely be more, as people find it fun to implement.
Gap buffer was appealing on really slow machines, where you are counting cycles on each key-press. If the gap is at the right position, the cycle count is very low. But you can probably do the same even with a tree-structure: you need to keep a pointer to the leaf in the abstract pointer used for the cursor.
Also I would tie in the undo system to the data structure if possible. Rope does this with copy-on-write. Every version of the file could be a different top-node, and most middle and leaf nodes would be shared between revisions.
The Craft of Text Editing
—or—
Emacs for the Modern World
–by–
Craig A. Finseth
which is more of a 10,000 foot view.
Theory and practice of text editors, or, A cookbook for an Emacs
They have a well architected model, including plugins to extend functionality, see https://tiptap.dev/ which is built ontop of Prosemirror
bonus: The author talks about collaborative editing here: https://marijnhaverbeke.nl/blog/collaborative-editing.html
12345|678
123
12345678
12345678
123|
12345678
12|3
12345678
123
123|45678
12345678
123
12345|678
This is definitely useful behavior. If you arrow down through a bunch of lines of various lengths, it's really annoying if the cursor starts drifting left. But implementing it correctly was much more subtle than I expected.
There are all sorts of other edge cases too. If a user presses Command-Up to move the cursor to the beginning of the file, and then presses Down, does the cursor always stay on column one, or does it remember the original column before Command-Up was pressed?
Architecture-wise, you can start with an ordered list of lines, with each line stored as a string.
Features that complicate things are:
- supporting large documents and staying speedy (“replace all” is a good test case)
- supporting line wrapping or proportional fonts (makes it harder to translate between screen locations and (line, character) offsets)
- supporting Unicode (makes it harder to translate between screen locations and (line, byte position) offsets)
- syntax-colouring
- plug-ins
- regular expression based search (fairly simple for single-line search _if_ you store each line as a string; harder for custom data structures, as you can’t just use a regexp library)
- supporting larger-than-memory files (especially on systems without virtual memory, but I think that’s somewhat of a lost art)
- safely saving documents even if the disk doesn’t have space for two files (a lost art. Might not even have been fully solved, ever)
Edit: you also want to look at https://news.ycombinator.com/item?id=11244103, discussing https://ecc-comp.blogspot.com/2015/05/a-brief-glance-at-how-...
Fair warning though, it's a fairly hard book to read. For a lighter, more fun intro to design patterns in particular, I always recommend Game Programming Patterns [2]
[1] https://www.amazon.com/Design-Patterns-Elements-Reusable-Obj...
This made selecting text quite hard. So I gave up and just put the whole thing into an array. It made everything a lot easier.
[1]: http://aosabook.org/en/index.html [2]: http://aosabook.org/en/eclipse.html
I know of The Craft of Text Editing.
https://www.finseth.com/craft/
There’s also the Racket editors, which includes a text editor control, in the Racket Graphical Interface Toolkit. It’s what is used to implement the DrRacket IDE.
1. General overview of the editor; maybe scroll to the "Implementation" section here: http://doc.cat-v.org/plan_9/4th_edition/papers/sam/
Some of the references at the end of that paper may also be relevant or interesting.
2. Tutorial for the command language: http://doc.cat-v.org/bell_labs/sam_lang_tutorial/sam_tut.pdf
3. And explanations on structural regular expressions that sam uses: http://doc.cat-v.org/bell_labs/structural_regexps/se.pdf
4. Pike's paper representing Acme, the (sort-of) follow-up editor of sam: http://doc.cat-v.org/plan_9/4th_edition/papers/acme/
Bracket pair colorization 10,000x faster - https://code.visualstudio.com/blogs/2021/09/29/bracket-pair-...
My editor had a modular architecture flexible enough to implement different input modes, including a mostly-complete subset of Vim bindings, and was fast enough to open and edit files on the order of a few hundred megabytes without perceptible slowdown. I'm sure my implementation would have looked insane to anybody who's worked on a real text editor, but I was fairly proud of it myself.
Anyway, I guess the point is that it was interesting and rewarding to navigate the core challenges myself. I'm not sure I would have gotten as much out of trying to understand how a massive project like vscode is put together, since the actual text editing functionality is (presumably) a comparatively small part of the software as a whole.
https://news.ycombinator.com/item?id=11244103
It’s about data structures for editable text. Surprisingly complex!
This is written in rust and has docs about rope data structure and editor architecture.
[1] DevTools hacking playlist - https://www.youtube.com/playlist?list=PLMOpZvQB55bfeIHSA71J8...
Even so, this is exactly the right thing to do, except probably you are studying editors that are too big for your purposes - find something smaller in scope.
You won't understand without significant effort - you need to put in the work.
So, here's what I suggest:
1: keep examining the source code of various editors - however, focus on trying to find small editors that are very focused in what they do. Also look for old editors for operating systems like DOS - they might be smaller in scope and therefore easier to understand. Also look at editors in other languages, such as Pascal.
2: The essence of learning is to implement - so start writing the smallest editor you can.
3: continue searching for and reading any sort of written documentation/blog posts/articles as you are doing.
Eventually the light will switch on.
as trying to write it yourself
and then reading about how other people do it e.g VS Code blog
https://code.visualstudio.com/blogs/2018/03/23/text-buffer-r...
There was a book explaining the implementation of this IDE:
https://www.amazon.com/Dissecting-C-Application-Inside-Sharp...
I think you might find it interesting for what you are trying to do, even though it's a bit old.
https://github.com/rxi/lite
https://rxi.github.io/lite_an_implementation_overview.html
https://rxi.github.io/a_simple_undo_system.htmlIf you're interested more in the complexity management--which I would call more software engineering-y--not sure what the best way might be! I know one cool thing about Vs code is the language server provider (LSP). This provides all the IDE-goodies for all the languages Vs code supports in an abstract interface any plugin can implement. The spec has been so popular it's now supported in a bunch of editors! So you could develop an LSP for eg python, and then just have a light wrapper for vim, Vs code, Emacs.
Note that it’s behind a paywall (his catalog is likely worth it for a curious hacker such as yourself).
Watch his “compiler from scratch” for free to get a sense of what his casts are like: https://www.destroyallsoftware.com/screencasts/catalog/a-com...
- https://arctype.com/blog/sql-query-editor-switch/
- https://arctype.com/blog/indexeddb-localstorage/
Disclaimer: I wrote the 2nd one. My general takeaway is that it gets interesting quickly as the amount of text edited increases.
https://sourceforge.net/p/joe-editor/mercurial/ci/default/tr...
This is a series that I used to follow as I too wanted to implement my own text editor to learn the underlying architecture.
There are a few weird design decisions especially in swing, but overall it's very easy to read and understand.