With this access you can read and write (and create,...) these "nodes". This makes it possible to completely circumvent TeX's input language, which can be a pain to program with (sometimes it is very nice though). Now that easy access to data structures and the rather nice (simple) programming language of Lua makes it easy to build rather complex programs on top of TeX.
A shameless plug is my database Publishing software (see https://speedata.github.io/publisher/index.html) which I believe takes this approach to an extreme: I can use TeX's algorithms like paragraph building, hyphenation, PDF writing and such (they are of excellent quality) and build my own software around it. I don't say that this is done on a weekend or so, but the posibilities are much, much, much greater than they were before LuaTeX was available.
Do you think such a thing would be possible with LuaTeX? I did glance over the article you linked and it's showing that it works pretty nicely for basic text, but can anyone shed some light if this works nicely for outputting equations, etc?
Depending on your need, LuaTeX can be quite fast. I have a project where I use a non-optimized version of LuaTeX to have a 'live' preview of a web form. It takes less than 1/2 second to run LuaTeX, generate a bitmap from the PDF and push it to the user. In most cases this will be too slow, but for me this was a good solution.
I really hope a big company doesn't pick Lua up, because I think the fact that it hasn't reached "Eternal September" yet is what makes it such a good language. It has room to breathe.
(The fact that it's beginner friendly, has an ANSI C89 implementation, and has the best/fastest JIT/FFI doesn't hurt either though :P)
I wonder if you can provide more background on that, particularly those last 2. And by AT&T, you mean Unix, right? :p
One of the reasons Lua's evolution is better than JS is because managing versions when you have to keep browser compatibility is way more difficult.
Both Lua and LuaJIT are maintained and in active development.
"an extended version of pdfTeX using Lua as an embedded scripting language" doesn't help me that much.
What's the advantage over (La)Tex?
There are currently three engines in use in the TeX community: pdfTeX, XeTeX and LuaTeX and they all serve an audience.
The pdfTeX engine is a stable extension to TeX that has eTeX on board as well as a few extensions. It provides hz and protruding and supports advanced PDF output. It is fast and reliable and sort of frozen with respect to functionality. It will stay around for ever but has only limited provisions for dealing with OpenType fonts.
Then there is XeTeX which supports Unicode as well as OpenType by means of third party libraries. It integrates nicely into the current infrastructure and support from macro packages is easy as there are no fundamental changes in interfaces and functionality. It uses the eTeX r2l typesetting while LuaTeX uses the Omega (Aleph) method. Support from macro packages does not demand changes in the core.
The LuaTeX project uses a different approach. Instead of using libraries it provides an extension mechanism. This keeps the program independent and permits the flexibility that we like so much. However, it comes at a price. First of all LuaTeX does not have the performance of its cousins, although for instance the average ConTeXt MkIV run can be faster in LuaTeX. LuaTeX also demands extensive support at the macro level if you want to profit from its benefits. Just adding some support for scripting is nice but the power of LuaTeX only shows when it's tightly integrated. This integration is also needed in order to provide macro users some stability as most of them are no TeX programmers. If such integration is not what you want, you might consider sticking to the other engines as there are no real advantages to using LuaTeX then.
Maybe this is something like what LuaTeX is trying to do? It's hard to tell from their web page.
TeX's input grammer can change during the TeX run, so I believe it is impossible to make an equivalent of LLVM (unless I misunderstand the concept of LLVM).
That is exactly what I am saying.
> TeX's input grammer can change during the TeX run, so I believe it is impossible to make an equivalent of LLVM
The important part is not the grammar, but the internal representation. I don't know enough about TeX to know much about its internal representation post-parsing.
Neither can C++[0], yet there still are C++ compilers out there. It does raise the bar significantly, though.
Unfortunately it doesn't have a (pure) TeX reader yet, but that could be implemented relatively easily.
BTW is there anyway to make LuaTeX compile faster?
I was amazed about the speed increase. (although it's kind of obvious in retrospect..)