Lisp enthusiasts like to point out the power of macros, and macros are the raison d'être for Lisp's homogeneous s-expression syntax. Most other features in Lisp (such as first-class closures and higher-order functions) can exist without s-expressions, but the powerful thing about s-expressions is that they enable Lisp macros.
But with great power comes great responsibility. When I'm writing a program I want as little responsibility as possible while still being able to solve the problem at hand. I don't want to be responsible for memory management and bounds checking, and I don't want to be responsible for the hygiene of my macros at both the definition and the call site.
With C, the responsibility of memory management and bounds checking comes with a power that people actually need to solve problems. For me these problems usually come up in the context of writing my own hobbyist interpreters/compilers, but there are a lot of real world cases where these come up. But often you don't need the capabilities of C, and I'd argue as a result that there are a lot of cases where using C is a bad idea because it's not the best way to solve your problem.
And here's the hot take: the power of Lisp macros isn't actually ever worth the responsibility in my experience. The problem Lisp macros solve is "this code is more verbose/ugly/boilerplate-y/etc. than I want it to be", which just isn't the problem you're writing the program to solve. Whenever you reach for a macro, there's another tool you could be reaching for to solve the actual problem at hand. At the very least, you can just write the code the macro would expand into. There's inherently never a case where the macro is the only way to solve the problem.
If you're good at writing macros, you won't always get burned by them, but nobody is ever perfect at writing macros, so everyone gets burned sometimes. If you're writing software that you actually need to work, the risk is rarely worth it.
When I have written production code in a Lisp (mostly Clojure) I've rarely reached for macros, and often bugfixes have been removing a macro that was of the "so preoccupied with whether or not they could, that they didn't stop to think if they should" variety. And if you spend enough time avoiding and removing macros, you start to wonder why you're destroying your eyesight trying to match parentheses, when the entire reason for the parentheses is to enable something you have to avoid and remove.
And don't get me wrong: macros are cool. "Because I like them" is a totally valid reason to write macros and Lisp.
Whenever you write code, you could just write the function it calls.
One purpose of macros is to create descriptive notations. That's one way to create very readable, maintainable code, ... Lisp itself uses macros in the language everywhere: to define functions, to define classes, to define namespaces, to provide control structures, ... the core special operators are at a minimum and the large amount of syntactic operators are written as macros. As a developer I can use the same power to shape the operators of my domain, beyond what functions offer me and with compile time efficiency & syntactic control/abstraction.
This is overly-glib. Calling a function doesn't come with the risk that calling a macro does, which I'm sure someone named "lispm" is aware of.
> Lisp itself uses macros in the language everywhere: to define functions, to define classes, to define namespaces, to provide control structures, ... the core special operators are at a minimum and the large amount of syntactic operators are written as macros. As a developer I can use the same power to shape the operators of my domain, beyond what functions offer me and with compile time efficiency & syntactic control/abstraction.
This just shows you don't understand the power/responsibility argument you're responding to.
As a developer I don't have the resources to design and test my code to the extent that the language creators do.
Lisp itself uses direct memory access to define the garbage collector, and you could use the same power to shape the memory usage of your domain, beyond what garbage collection offers and with compile time efficiency. But surely we can agree that's a terrible idea in most cases.
Expand your experience.
> At the very least, you can just write the code the macro would expand into.
You can "just" get it wrong when the code is used in many places.
Consider this as the hot take: macros reduce the number of characters you have to type. I haven't seen anything that beats macros in this metric.
Macros allow writing code that's both compact and readable [0], in lesser languages you have to choose.
[0] https://github.com/codr7/cl-redb/blob/ff3a34a31ced7a9668fc95...
[1] https://gist.github.com/codr7/4bb9442c0c66411643eddd8db0164a...
That's universally good advice, which you should follow too.
But neither of us is going to experience all that exists to experience, so it makes sense to prioritize. I've experienced enough pain debugging Lisp macros to make an educated decision to deprioritize further exploration in that area.
> You can "just" get it wrong when the code is used in many places.
True, which is why I generally use other abstractions to avoid repeating myself.
> Consider this as the hot take: macros reduce the number of characters you have to type. I haven't seen anything that beats macros in this metric.
Sure, but within reasonable languages that's not a metric which matters. Typing has never been the bottleneck of my development. Even in absurdly verbose languages like C++, IDEs generate a lot of the code for you: the bottlenecks in C++ are things like stitching together different ways of doing the same thing which were used in libraries or different parts of the codebase.
Do you mean to say that you were intent on taking a good look at Janet when you woke up this morning, but because of this post specifically your mind has been radically changed?
Manually matching parentheses is a non-issue since tools like Parinfer were created: https://shaunlebron.github.io/parinfer
(defmacro ->
"Threads the expr through the forms. Inserts x as the
second item in the first form, making a list of it if it is not a
list already. If there are more forms, inserts the first form as the
second item in second form, etc."
{:added "1.0"}
[x & forms]
(loop [x x, forms forms]
(if forms
(let [form (first forms)
threaded (if (seq? form)
(with-meta `(~(first form) ~x ~@(next form)) (meta form))
(list form x))]
(recur threaded (next forms)))
x)))My fav example is automatic sql table and column typo checking. The macro introspects the db and you get syntax errors if the table/column isn’t found. Same idea can also be used for csv and json. Or api spec file on different server-the compiler can make http requests to show you syntax errors
But the problem in all these cases isn't "I want syntax errors if this table/column doesn't exist or my http request doesn't fit spec", it's "I want to make sure my table/column exists and my http requests are in spec". Unit tests can do that. What unit tests can't do is go into the code under test and modify it so that it breaks in production. The macros you're describing can do that.
The macros you're describing are cool, but they don't do anything that can't be done with unit tests, and they're not without risks. You're essentially running test code in production.
That's a curious statement, given that the stuff discussed doesn't exist in Janet.
> This post persuaded me to not take much of a look at Janet.
I feel I should say that you shouldn't form too much of an opinion about Janet just because one person used it to prototype a weird macro system. This concept is not, like, part of Janet or anything -- Janet macros are basically Common Lisp's, but with an elegant solution to the function hygiene problem -- and I don't think that I'm even a very representative Janet user (as a big macro fan). Janet is a very nice Lua and Perl alternative even if you never use it to write a single macro! Its text parsing facilities alone are worth a look.
> The problem Lisp macros solve is "this code is more verbose/ugly/boilerplate-y/etc. than I want it to be", which just isn't the problem you're writing the program to solve.
I think this misses what I see as "the point" of macros, which is to be able to make a tiny language core. Consider "and:" I would be sad to program in a language without a short-circuiting "and." So most languages special-case that, right? But lisps don't. Macros mean that you don't have to make "and" a built-in part of the language. Or, I dunno, "defn." "for." Janet's only iteration primitive is "while," and then standard library uses macros to implement for, each, list comprehensions, etc.
And I feel like it's totally fair to not care about that all. After all, why does it matter to you, the programmer, whether "and" is special-cased in the language what is implemented as a macro in "user space?"
> Whenever you reach for a macro, there's another tool you could be reaching for to solve the actual problem at hand. At the very least, you can just write the code the macro would expand into. There's inherently never a case where the macro is the only way to solve the problem.
Of course! But I feel like this doesn't make a compelling point against macros to me, because you could say exactly the same thing about first-class functions, or generic types, or any other language feature. And people have!
> If you're good at writing macros, you won't always get burned by them, but nobody is ever perfect at writing macros, so everyone gets burned sometimes. If you're writing software that you actually need to work, the risk is rarely worth it.
This is, I think, a very good point. Which is why lots of languages (Racket, Scheme, Clojure) have macro systems that make it almost impossible to write macros that "burn you," if I understand your meaning correctly. But others don't! Janet is a bit weird in that it's a recent language that does not have a hygienic-by-default macro system.
> When I have written production code in a Lisp (mostly Clojure) I've rarely reached for macros, and often bugfixes have been removing a macro that was of the "so preoccupied with whether or not they could, that they didn't stop to think if they should" variety. And if you spend enough time avoiding and removing macros, you start to wonder why you're destroying your eyesight trying to match parentheses, when the entire reason for the parentheses is to enable something you have to avoid and remove.
I can certainly see how such an experience would sour you on macros. I am fortunate that I have never had to maintain buggy legacy macros -- sounds awful. I would again point out that you could have a similar experience with... inheritance, async/await, static type checking, I dunno. Anything applied poorly can seem like a terrible idea. You could write off entire languages if you only saw terribly written code in that language. But macros applied well can be great!
When I think of all the language features that have been added to, say, JavaScript over the last twenty years, and how many of them could have been written as macros, in a way that works for all browsers, without a need for something like Babel... it's a little silly that JavaScript developers had to wait for async/await to become an official language feature, when Clojure just implemented it as a library.
Counterpoint: it's reasonable to argue that it's bad to add language features to a language, because it makes your code harder to understand for the median developer. But of course the lack of macros doesn't stop language fragmentation, it just relegates that to a smaller group of programmers who have the time or inclination to write full parsers and compilers -- see JSX, Svelte... Clojure itself! Or Kotlin, or any other JVM language.
(Not that macros prevent fragmentation. If anything, the fact that macros make it so easy to implement a lisp is why we have so many different lisp implementations...)
> And don't get me wrong: macros are cool. "Because I like them" is a totally valid reason to write macros and Lisp.
Yeah :)
I wanted to respond to something else you said elsewhere:
> As a developer I don't have the resources to design and test my code to the extent that the language creators do.
I think that blurring the line between "code users can write" and "code language authors can write" is the point! To give programmers the resources to design and test code on the same level as language implementors. (Which, again: why should I care?)
Sure, the alternative to hygenic macros was what originally made me curious about that, but now I've learned enough about that to satisfy my interest. However, I'll retract my statement that I'm not going to look into it further, because I had forgotten I was also interested in how they get it to embed in C programs.
> Of course! But I feel like this doesn't make a compelling point against macros to me, because you could say exactly the same thing about first-class functions, or generic types, or any other language feature. And people have!
Right, but the complete argument isn't "there are other ways to do this", the argument is "there are other ways to do this, and nearly every one of them is less error-prone".
> When I think of all the language features that have been added to, say, JavaScript over the last twenty years, and how many of them could have been written as macros, in a way that works for all browsers, without a need for something like Babel... it's a little silly that JavaScript developers had to wait for async/await to become an official language feature, when Clojure just implemented it as a library.
I'll point out that JavaScript already had async callbacks and promises (implemented as a library) when async/await was added as a third way to do basically the same thing, resulting in JS codebases that now have half-baked glue to make the three ways work together. I'm not sure anybody was waiting for async/await to do anything that couldn't already be done, and the churn of reimplementing working code to use a new feature didn't do anyone much good. But that's sort of a tangent.
> Counterpoint: it's reasonable to argue that it's bad to add language features to a language, because it makes your code harder to understand for the median developer. But of course the lack of macros doesn't stop language fragmentation, it just relegates that to a smaller group of programmers who have the time or inclination to write full parsers and compilers -- see JSX, Svelte... Clojure itself! Or Kotlin, or any other JVM language.
Language fragmentation is a problem, but it's not the problem I'm talking about.
If I write code in a popular programming language such as Python, JavaScript, C++, Clojure (sans macros), etc., I create bugs, and I can be reasonably certain that those are my fault. I've been writing C longer than anything, and I've never found a bug in GCC or Clang in over 20 years (okay, there are a few that were retroactively declared features and forever-supported, but that's a separate issue). As The Pragmatic Programmer says, "Select isn't broken", and Coding Horror says, "It's always your fault"[1]. It's not necessarily rare for a popular programming language to have bugs, but it's extremely rare that you'll be the first one to find them.
The same is true for popular libraries and whatnot that ship with the language, which is why I'm not particularly concerned about "defn" or "for" are macros. Those are macros in most implementations, I'm aware, but they're really-well-tested macros because pretty much every Lisp developer to ever write a significant amount of Lisp has tested them. "defn" and "for" aren't broken.
If I write code in the half-baked DSL written by Bob two cubicles over using macros. That code definitely has bugs, and it's very likely I'll be the first to find them. Not to hate on Bob too much: if I wrote macros they'd have bugs too.
And sure, as you've said, Bob can write buggy functions too. The difference is, functions and I have really good boundaries. When I call a function it doesn't touch my code, and I don't touch it's code, and the expressions I pass into Bob's functions only get executed once, and the stack traces all have very understandable corresponding line numbers, and most of the time it's very easy to figure out if the bug is in Bob's function or my code calling Bob's function. And if it's in Bob's code I write a unit test and fix it, and if I'm feeling cheeky I send him a screenshot, and if it's in my code I fix it and git rebase my mistake out of existence to hide my shame.
Macros don't have those boundaries. The expression I pass as an argument to a macro might get called once, twice, ten times, or not at all, with any side effects of that occurring each time. Symbols might get leaked. If you pass (+ (* m x) b) into a macro it can do stuff like flatten a parent s-expression too far make that into (+ * m x b) and even that simple issue can be hard to debug because the line numbers get split up so you have to figure out what's going on. So you can't really tell whether the problem is in the macro or the code calling the macro. And you don't even know when you have to be careful about this, because it's not always obvious whether the code you're calling even is a macro.
Hygenic macros do help, but they don't eliminate all of these problems.
And half the time when I git blame, it wasn't even Bob who wrote the macro, it was me, five years ago. Ain't that embarrassing.
> I think that blurring the line between "code users can write" and "code language authors can write" is the point! To give programmers the resources to design and test code on the same level as language implementors. (Which, again: why should I care?)
And that's my point: macros don't give you thousands of programmers to test your language you made out of macros. So that's why I care whether it was me or Bob or the Common Lisp team who implemented the language: when the Common Lisp team implements the language it doesn't matter if they use macros or assembly because thousands of people will run the code before I even get a chance and they'll suss out the vast majority of the bugs and issues before I have to deal with them. When me or Bob implements the language, it's me, Bob, or the intern who has to suffer the consequences.
Codebases that use extensive macros to create DSLs eventually become write-only. The power and readability you see in toy examples and in the short run in your own code, rarely plays out in the long run, and when it does play out it's because of extensive testing and work--a lot more work than Bob and I have the bandwidth for.
[1] https://blog.codinghorror.com/the-first-rule-of-programming-...
LISP has a full-featured macro system, thus hands down beats many languages that only possess handicapped macro system or no macro system at all. It uses the same/similar language to achieve it is mere accidental. In fact, I think LISP is an under-powered programming language due to its crudeness. But it's unconstrained macro system allows it compensate the programming part to certain degree. As a result, it is not a popular language and it will never be, but it is sufficiently unique and also extremely simple that it will never die.
What if, we have a standalone general-purpose macro system that can be used with any programming languages, with two syntax layer that programmers can put on different hat to work on either? That's essentially how I designed MyDef. MyDef supports two forms of macros. Inline macros are using `$(name:param)` syntax. Block macros are supported using `$call blockmacroname, params`. Both are syntactically simple to grasp and distinct from hosting languages that programmers can put on different hats to comprehend. The basic macros are just text substitution, but both inline macros and block macros can be extended with (currently my choice) Perl to achieve unconstrained goals. The extension layer can access the context before or after, can set up context for code within or outside, thus achieve what lisp can but using Perl. We can extend the macros using Python or any other language as well, but it is a matter of the extent to access the macro system internals.
Inline macros are scoped, and block macros can define context. These are the two features that I find missing in most macros systems that I can't live without today. Here is an example:
$(set:A=global scope)
&call open_context
print $(A)
print $(A)
subcode: open_context
set-up-context
$(set:A=inside context)
BLOCK # placeholder for user code
destroy-contextI sorta agree. The simplicity of the syntax and representation makes it particularly amenable to dynamic modification above basically every other language though. There's a bunch of other properties that make this to case though, such as a very dynamic type system, first-class functions, extremely simple syntax, etc. so it's really a combination of a bunch of factors.
That being said, I think homoiconicity is actually a useful feature, but runtime macro expansion is the dangerous part. What I'd really like to see is a Lisp with very well defined execution orders. That is, macros must be expanded at compile time, and with clear symbols for defining what runs and when. I'm not talking about something like `(macroexpand ...)`, more like `(comptime (my-macro ...))`... `(define (my-func) (my-macro+ 'a 'b 'c))` where it's explicit that a macro executes at compile time, and usage of that macro must be denoted with a special character (`+`, in my example).
I think a general purpose macro language is only as useful as the average code-gen/templating language. It's just string manipulation, which can be harder to reason about than true metaprogramming and might result in some ugly/verbose output and having to context-switch between 2 entirely different languages often. What's particularly powerful about Lisp macros is that usage of them doesn't look any different than a normal application usually, and writing a macro is only marginally different than writing a normal function.
I would like you to reconsider. Predicting a program output is hard. So in order to comprehend a macro programed as code, one need run the macro in their head to predict its output, then they need comprehend that output in order to understand the code. I think that is unreasonable expectation. That's why reasonable usage of meta-programming is close to templating where programmer can reason with the generated code directly from the template. For more higher-powered macros, I argue no one will be able to reason with two-layers at the same time. So what happens is for the programmer to put on his macro hat to comprehend the macro, then simply use a good "vacabulary" (macro name) to encode his comprehension. And when he put his application programming hat, he takes the macro by an ambiguous understanding, as a vocabulary, or some one may call it as a syntax extension. Because we need put on two hats at different time, we don't need homoiconicity to make the two hats to look the same.
Practically, why would you ever want a runtime macro expansion?
You could have a replacement for M4 there.
I made a macro preprocessor in 1999 called MPP which also was able to preserve whitespace in perpetrating multi-line expansions, and so suitable even in indentation-sensitive formats. MPP has scopes and also a namespace system. I didn't develop it further because right around that time, I discovered Lisp and, you know ...
What the lisp approach gives you is the macro has the parsed representation of the program available to inspect and manipulate. It doesn't have to expand to some fixed text, it can expand into different things based on the context of the call site.
Various languages have decided that syntactically distinguishing macros from functions is a good idea. Maybe the function is foo and a macro would be $foo. For what it's worth I think that's wrong-headed, and the proper fix is to have 'foo' represent some transform on its arguments, where it doesn't matter to the caller whether that is by macro or by function, but it's certainly popular.
I can define any macro lisp can define, just not with LISP syntax. I do not have a full AST view of the code, due to the its generality that it does not marry to any specific underlying language. But I can have extensions that is tailored to specific language and do understand the syntax. For example, the C extension can check existing functions and inject C functions. MyDef always can query and obtain the entire view of the program, and it is up to the effort in writing extensions to which degree we want macro layer to be able to parse. Embedding a AST parser for a local code block is not that difficult.
It's like the innerHTML thing, for me, I always find the text layer (as string) is more intuitive for me to manipulate than an AST tree. If needed, make an ad-hoc parser in Perl is often simple and sufficient, for me at least.
This is my new favorite typo:
(defmacaron . lefts [key & rights]
~(,;(drop-last lefts)
(get ,(last lefts) ,(keyword key))
,;rights))
I use Kernel a lot, which allows you to write first-class operatives which can influence the bindings of their caller, but they don't allow modifying the body of the calling function.
For slightly better performance, there's bronze-age-lisp, which uses klisp and x86 assembly. Mirror: https://github.com/ghosthamlet/bronze-age-lisp
Performance will never be great due to the nature of the language, which is incompatible with usual forms of compilation.
I'll try to give a brief explanation of wrap and unwrap.
A combination is of the form `(combiner combiniends)`, where combiner must be either an operative or applicative. In the case that it is operative, the combiniends are passed verbatim to the operative, without being reduced. If the combiner is applicative, the combiniends are reduced, by evaluating each item in the list using the metacircular evaluator, until a list of arguments is returned. The arguments are then passed to the underlying combiner of the applicative.
Operatives are constructed using `$vau`, and applicatives by using `wrap` on another combiner. Usually the underlying combiner is operative, but the language used in the report is clearly permitting you to wrap other applicatives too, so in the case that you evaluate an applicative whose underlying combiner is applicative, and the underlying combiner of that is operative, then the list of combiniends would be reduced twice before being passed to the final operative. I've honestly not encountered a single use-case for this in my time using Kernel, but who knows. It might just be easier to consider that `wrap` wraps operatives into applicatives.
The description of the evaluator from section 3 of the report is a pretty clear explanation of what happens.
* If the expression to be evaluated is a pair, then:
* The car of the pair must be a combiner
* If the combiner is operative, call the operative with the cdr of the pair
* If the combiner is applicative, evaluate cdr of the pair to produce an argument list 'd. eval the cons of the the underlying combiner of the applicative with 'd.
($define! eval
($lambda (o e)
($if (not (environment? e)) (exit))
($if (pair? o)
($let ((c (car o)))
($if (operative? c)
(call c (cdr o) e)
($if (applicative? c)
(eval (cons (unwrap c) (eval-list (cdr o) e)) e)
(error "not a combiner in combiner position"))))
o)))
($define! eval-list
($lambda (l e)
($if (null? l)
()
($if (pair? l)
(cons (eval (car l) e)
(eval-list (cdr l) e)
(error "operand to applicative must be a list"))))))> So people have spent a lot of time thinking about ways to write macros more safely – sometimes at the cost of expressiveness or simplicity – and almost all recent languages use some sort of hygienic macro system that defaults to doing the right thing.
> But as far as I know, no one has approached macro systems from the other direction. No one looked at Common Lisp’s macros and said “What if these macros aren’t dangerous enough? What if we could make them even harder to write correctly, in order to marginally increase their power and expressiveness?”
The first example discussed is a defer macro that can be invoked with "no indentation increase and no extra nested parentheses".
As a macro-lover and an indentation-hater, I think this is a brilliant and hilarious idea.
(Just from a quick skim of this long post.)
[1] Macros for Domain-Specific Languages https://par.nsf.gov/servlets/purl/10220787 https://docs.racket-lang.org/ee-lib/index.html [2] From Macros to DSLs: The Evolution of Racket https://drops.dagstuhl.de/opus/volltexte/2019/10548/pdf/LIPI... [3] PADL'23 Modern Macros https://www.youtube.com/watch?v=YMUCpx6vhZM
You make a macro called crazy-macrolet which is used like this:
(crazy-macrolet ((crazy-macro (left-forms right-forms arg ...)
....)
(other-crazy-macro (...)))
body)
(defmacro bobs-crazy-dsl (&rest forms)
`(crazy-macrolet ((bobs-crazy-macro ...))
,@forms))
crazy-macrolet needs to implement a code walker to in order to expand those macros.
Macros can be context-dependent without having access to the literal forms to the left or right (or elsewhere).
In TXR Lisp, I implemented tagbody as a macro, providing a measure of CL compatibility. The go operators are local macros. They do not expand in a context-free way; they communicate with the surrounding tagbody.
So for instance if go is asked to jump to a nonexistent label, it errors:
1> (expand '(tagbody (go a) b))
** go: no a label visible
1> (expand '(tagbody (go a) a))
(let ((#:tb-id-0019
(gensym "tb-dyn-id-"))
(#:next-0020
0))
(sys:for-op ()
(#:next-0020)
((sys:setq #:next-0020
(block* #:tb-id-0019
(sys:switch #:next-0020
#(((return* #:tb-id-0019
1))
()))
())))))