Smaller Code, Better Code (opens in new tab)

This means that in those cases where reuse is valuable, it's very valuable, and it comes to the fore and you can see it as the critical thing that it is. It doesn't get drowned in otherwise petty abstractions that assist reusability, since we don't need that anymore.

Furthermore, if I write my code correctly, there is very, very little boiler plate in the compiler. Almost none. This means that every line is significant. By doing this it means that you don't get the fun of feeling like you're accomplishing something by typing in lots of excess boiler plate, but it does mean that you have no wasted architecture. Because rewriting the architecture is so trivial, basically everything now becomes important, and you don't have petty book keeping code around. You know that everything is important, and there is no superfluous bits.

The result, as mentioned elsewhere, is code that is getting continuously simpler, rather than continuously more complex. The code is getting easier to change over time, not harder. The architecture is getting simpler and more direct and easier to explain. Because it costs so little to re-engineer the compiler, I can do so constantly, resulting in little to no technical debt.

This is an intentional synergistic choice of a host of programming techniques, styles, disciplines, and design choices that enables me to program this way. Give up one of them and you start to break things down. It allows for a highly optimized programming code base that has all of the desirable properties people wish their code bases have, and it scares people. I think that's a good thing. Because I don't want people to see this codebase as just another thing. I want them to see that this is something truly different. How can I get away with no module system? How can I get away with no hierarchy? How can I get away with having everything at the top-level, with almost no nested definitions? How can I get away with writing a compiler that is not only shorter, but fundamentally simpler from a PL standpoint than standard compilers of similar complexity by using only function composition and name binding? How can I get a code base that has more features but continues to shrink?

By chasing smaller code. :-)

I assure you, and I'll make good on this in another reply here, I could get you up and running on understanding the code and how it works faster than just about any other compiler project out there. In the end, one of the goals I want for this compiler is for people to say, "Woah, wait, that's it? That's trivially simple." The more I can push people to think of my compiler as so trivial as to be obvious, the more I win. The compiler really is so dirt simple as to shock any normal compiler writer.

But to make it that simple, I have to do things in ways that people don't expect, because people expect complexity and indirection, they expect unnecessary layers for "safety" and they expect code that needs built in protections because the code is too complex to be obviously correct.

I'm pushing the other direction. If you can see your entire compiler at one go on a standard computer screen, what sort of possibilities does that open up? You can start thinking at the macro level, and simply avoid a whole host of problems because they are obviously wrong at that level. When you aren't afraid to delete you entire compiler and start from scratch? What sort of possibilities does that open up to you?

That's a great counterargument, and one I fully agree with. I've noticed that over the years, there has been a growing trend of promoting "readable, maintainable, clean, insert-fashionable-adjective-list-here code" which really amounts to a lower-common-denominator, dumbing-down perspective of how software should be written. In their perspective, code that someone does not immediately understand is "bad", seemingly regardless of how much (or little) knowledge that someone possesses. I think this is ultimately a harmful trend.

The opposing view, which appears to be largely a minority in more mainstream language communities but dominates in others like APL and Asm, is that programming languages are essentially like human languages: they need to be learned, are not necessarily "easy" or "familiar", and this learning and eventual mastery is wholly beneficial to their use. As with human languages, it is not expected nor a problem that a beginner will immediately understand code written by a more advanced user. Instead, the beginner progresses by learning the language and eventually becoming an advanced, "literate" user. This can be summed up in one sentence: "The code is unreadable because you are not yet qualified to read it." ;-)

I am perfectly willing to believe that I could reduce the size of my code by a factor of 10, maybe even 100, if I was willing to give up the constraint of making it maintainable independently of myself. I think that would be a poor tradeoff to make in most cases.

What additionally interests me is the combination of points-free style and the kind of data structures you're processing in an array-biased language, could you give an insight on what that is like to work with?

In particular, I presume from your description, and only a conceptual familiarity with APL, that most or all of this code is "functional", i.e. all data structures exist as values passed between the composed functions, and nowhere else (no globals or similar). I'd love to hear more about the predominant data structures and what shape they take.

Somewhere else you mention Quad-XML, which seems to be a way to represent trees as arrays, with each element pre-fixed with its depth. I presume you use this for the AST? What kinds of operations are simpler on these arrays, and which are harder, compared to tree data structures used in other languages? For example, addressing the Nth child from a parent could be harder, since you have to search past the other children? I could imagine that operations like "set all fields X of the tree to Y" are a lot easier since no tree traversal is required.

Does your ability to quickly refactor rely on this functional nature?

This is argument from analogy, and with a certainty nearing 100% it doesn't apply to programming languages.

If you want to take this argument all the way though... Why not use Japanese instead?

It's ugly, it's unreadable, it takes countless hours to master the language, the grammar, the writing system. In the end you arrive... at yet another language[1]. Which may or may not express some things that English can't. By the time you've mastered Japanese, you'll have achieved near perfection and all your goals in English :)

[1] I speak four and currently am in the process of learning a fifth language (Russian, Romanian, Turkish, English, Swedish). I can say with some "expertise" that you can't make direct comparisons between natural and computer languages.

imagine being presented with this and tasked with maintaining this. or adding a language feature. i'm certain the author could do it without much effort, but this code is as short as to be obfuscated - i have had more understanding from ioccc entries than this.

code exists as a common language for humans to understand and collaborate. this code is nightmare-ish.

Firstly, about this being a personal project, it's actually a bit more than that. It serves as a research platform for a research agenda around the usability of programming languages and the HCI and pedagogy of computation, yes. However, it's also a commercially funded compiler that is commercially licensed and distributed. The compiler is still in early stages, so it's large a boutique offering at this point, but that's scheduled to change this year or maybe the next. And yes, the development team that works with me on this compiler has read the code, and while they are not as fluent in it as I am, they understand how to work with it and we can talk about the compiler and work through issues in the compiler that comes up.

Indeed, the fact that the compiler is so easy to track through at a macro level has allowed us to avoid needing extra documentation throughout, because when we have a question about some level of architecture design, we can usually pull up a page of the compiler and work through it without needing any other documentation.

And, the point of the post above was that small code is a useful metric for pushing for simplicity. There is a difference between obfuscation and small code, but my code is not obfuscated to those who need to work with it. It is obfuscatory for anyone who expects to read it like a normal program.

At the heart of this is the meaning of readability. You've implicitly defined readability as being a state for code bases that allows anyone else to read and understand the code. That's a high bar. If I write a standard proof of the uncountability of the real numbers, it's a rather high bar to say that everyone should be able to read that proof.

Also, if you look at the way that the Clang codebase is engineered, for instance, if you take any one snippet of 50 lines or so of code, it's all nice, neat, and readable. But when it comes to understanding the entire compiler as a whole, the codebase is completely unreadable. It requires external documentation to understand almost any part of that code at a macro level.

But Clang uses best industry practices and is, on the whole, what most people would consider very cleanly written code. And yet, it is essentially impenetrable from a macro level without other documentation.

Instead, I'd submit that readability is something that we should consider valuable for those who have the relevant pre-requisite understandings of key ideas and concepts when looking at a new code base.

Part of the problem is that this code base is introducing new, research level ideas into the coding space. There is a fundamental difference between it and other compilers in the approach that it is taking, and thus, you can't just look for the same patterns.

I've already touched on malleability elsewhere. SICP has a classic quote about the importance of malleability in code (amoeba vs. pyramid programming).

I'll give a simple description of the architecture. If you understand everything said in the following sentence, then you'll have no trouble understanding how the code is written, and if you don't, then learning these sub-domains of programming skillsets will go a long way in helping to clarify the design.

It's a three part dfns->C++ offline batch compiler overloading the standard Quad-Fix system function in APL interpreters to compile whole, closed namespace scripts built of pure functional dfns on the Dyalog 15.0 primitive vocabulary sans-guards through a PEG parser to a core compiler written in a Nanopass compiler architecture over a linearized Quad-XML style matrix AST representation where each pass is written as a data-flow, data-parallel function train leading to a single dispatch code generator with a runtime library header prepended to each output file containing implementations of each implemented APL primitive.

Those would be the basic set of techniques and skills that are being put to use in the compiler. If you already understand Nanopass, PEG parsers, the Quad-XML tree linearization format, function trains, and so forth, then the structure and format and design of the compiler is obvious and easy to work with after about 5 minutes of orientation. If you don't have that background, then understanding that part of the compiler is rather a difficult one. In addition to this, there are new techniques being used and applied to solve problems in this compiler itself, and those are being documented through the papers that I'm publishing on these techniques:

https://github.com/arcfide/Co-dfns#publications

Most people don't have a strong data-parallel, array style programming background, which makes the micro-level code the hardest part to understand for them. However, if you are experienced in that background, then working with the compiler passes is not difficult, provided that you take the time to understand the core idioms in play.

So, in summary, I'd say that you're right that the code looks horrendous, because your heuristics are designed for code that is completely, almost assuredly, fundamentally different than this code. However, like I said, come to the live session and see me explicate the architecture of the compiler. I'll explain a lot of the ideas I mention in the above sentence enough to allow you to walk through the code easily. If you still think it's scary, okay. I'd appreciate some ways to make it easier to work with it on a day to day basis.

The use of single letter names in the files here is a bit of an inside joke, referencing back the style of programming of Arthur Whitney, signaling a bit of a historical "stylistic" or artistic connection, while at the same time being the first "alert" to the programmer that they are likely to see something along the lines of Whitney style C code inside of the files. It serves both as a chuckle to the APL community as well as a documentation of how you might want to prepare your mind before reading the code.

Please omit such offensive/defensive rhetoric from your posts to HN. It adds no information and is bad for conversation.

The problem here isn't "daring" to criticize, it's rejecting the unfamiliar. This is like traveling to a new country and complaining because they cook everything wrong and say everything wrong. Unfamiliarity is relative—it's not a property of the thing you're reacting to. Same with readability: it's relative to the reader.

In some contexts this is obvious. If you don't know German, you wouldn't reject a German text as unreadable or poorly written. But in other contexts, when we unconsciously assume or were taught that there's only one valid way to do something, we react with shock and distaste at work that violates known conventions. Such work may in fact be organized around different conventions for reasons we don't yet see. Good conversation across such boundaries requires a bit of distance from our own assumptions.

Programming is like the world of art this way. There are countless examples in art history of sharp departures from convention provoking shock and distaste, and people saying things like "There's a reason why readable and beautiful [art] is favoured". Riot police famously had to be called to the early shows of the Impressionists, yet the beauty of their paintings is obvious to us now.

A dense expression can still take a while to puzzle out sometimes, but certainly no longer than the equivalent logic spelled out in a more verbose language across many lines.

APL really encourages code golfing. It's barely readable in the best of circumstances, so there's a lot of temptation to compress code. Er, it's readable, but it's like reading a regular expression. You read it character by character, not line by line.

The other thing APL encourages is writing dimensionality independent code. Just as a good C programmer will write a function to concatenate two strings that works independent of the strings' sizes, a good APL programmer will write a function to accept arrays of arbitrary dimensionality where that can make sense. That's because it was not uncommon to have intermediate results of 3, 4 or 5 dimensions. Also, most of the built-in operators do something useful with higher-dimensioned inputs.

So yes, it becomes second nature, more or less. But at the same time, it restricts the kinds of problems you think about solving with it.

Today I would not recommend APL for any purpose except studying its place in computing history.

Never seen that. Thanks for sharing as it might be a great way to drive point home to business types. The amount of code certainly turns [for developers] from asset into a liability as it's size and usage grows. Hmm. Maybe such a presentation could always consider the amount of code a liability or neutral asset that produced benefits on the other side or reduced them. A well-stated connection between the two might justify reducing technical debt.

In my opinion it is the number of requested features shipped minus the number of bugs introduced. Weighted by the importance of each, as collectively decided on by everyone or client.

One more point: I find that there are a lot of very common things that we, as developers, have not 'standardized' on - but if we did, it would be beneficial.

The underscore/lodash JS libraries are great examples of this.

They are not just a bunch of 'helper functions' - they are really a series of new 'functional keywords' that in a way represent a new paradigm in software: we all get used to these 'mini patterns' and call them the same thing, and when used in code they can make things a lost simpler.

Map, reduce, find, each, pull, filter etc. etc. - at first glance it would seem compulsive to jam all these into some code - but once the developers are familiar with them ... guess what - they become almost part of the programming language itself.

So I think this is a pretty good example of a 'meta' way to facilitate simplicity: agree on names for very common patterns, and abstract them away with tools or linguistic constructs.

This compiler has gone through many core paradigm shifts in an attempt to find an appropriate way to express a solution to the problems that it encountered. Each iteration revealed some new insight into how to solve the problem, but inevitably lead to a need to rethink the system.

Now, the system is so expressive and capable that reusability isn't even an issue. At this point reusability is about as useful in the compiler as having a new word to represent the word "the". Why? Why not just write the? Anything else you could write is likely to create confounding layers of indirection and distance between definition and use in the code that will actually obscure clarity.

Instead, I take the intentional approach to make the code as "disposable" as possible. Why change a compiler pass that is two lines long when you can just rewrite it from scratch in less time? By leveraging a different aesthetic, architecture, and language, I'm able to have more expressivity by removing unnecessary abstraction and making it as easy as possible to re-engineer the whole thing at the drop of a hat. This means that I never have to "live with" code bloat or some design decision that's annoying me. The cost to re-engineer is so low that I have almost no technical debt. If an architecture fails to scale, replace it and move on, without any loss of productivity, and a net gain since the code gets easier and easier to work with on each iteration.