"A smart programmer is not necessarily an empathetic language designer; they are occupations that require different skillsets. Giving any programmer on your team the ability to arbitrarily extend the compiler can lead to a bevy of strange syntax and hard-to-debug idiosyncrasies."
There are at least 2 counter-arguments to this:
1.) in the simplest case, just restrict the use of macros. A team can easily adapt the rule that only the most experienced engineer on the team is allowed to write or approve macros. (And in my experience, the need for macros is fairly rare. I think my ratio is something like 100 or 200 normal functions for every macro that I write.)
2.) macros allow all kinds of interesting type checking, and data structure validation, and therefore they make it surprisingly easy to validate data and types as your data and/or vars get passed around your system. Consider all of these very interesting tools you can use in the world of Clojure:
Prismatic Schema which allows validation that a data structure matches a schema of (possibly nested) types:
https://github.com/prismatic/schema
and this now offers coercion, which makes this fantastic for importing JSON from other sub-systems or outside vendors:
http://blog.getprismatic.com/blog/2014/1/4/schema-020-back-w...
(I assume you could easily validate before giving data to Liberator to export your data while conforming to your schema: http://clojure-liberator.github.io/liberator/ )
There is work being done on an optional type system:
https://github.com/clojure/core.typed
Much effort has been made to make contract programming easy in Clojure:
https://github.com/clojure/core.contracts
But also I find the built-in syntax for writing pre and post assertions is clean and easy to use:
http://blog.fogus.me/2009/12/21/clojures-pre-and-post/
In short, there are an abundance of mechanisms available with Clojure which help facilitate the enforcement of any kind of schema or contract, and some of these tools are enabled (and their syntax is made clean) thanks to macros.
In short: macros can be used for evil, but they can also be used for good. They are very powerful, so everyone should be judicious about their use, but there is no reason to argue that macros render a Lisp unfit for programming in the large.
Having said all that, I'll remind everyone that the ultimate counter-argument is offered by Paul Graham, in his essay "Beating the averages":
http://www.paulgraham.com/avg.html
If that essay does not convince you of the value of macros/lisp, then nothing will.
I think there's a kernel of a valid point underneath. A macro necessarily has a larger (potential) interface surface than a function, because there's less you can take for granted about how it interacts with the rest of your code. And I agree, API design is a specialized skill that requires quite a bit of thought.
But I agree with your conclusion more than his: This is to take care with macros, not a reason to shun them altogether.
Precisely. The problem with the lisps of old was entirely cultural. People would go do crazy wild things and then not bother to interoperate with the rest of the world. Meanwhile, the Clojure community has lots of experimentation, but ultimately produces a large number of stable, quality, reusable libraries.
Large scale C++ teams often require approval for operator overloading or "dangerous" features. Can easily do the same for macros. Moreover, we now have distributed version control and can utilize lieutenant workflows, so we can dispatch with this silly "commit bit" notion that means bad code sneaks in past domain experts with ease.
* a teaching language
* a research tool
* an application programming language
Lisp has been already in times when the technology you are using today was still under invention. Lisp existed before Smalltalk, C, C++, Java, ... thus often technology was developed in an unstable surrounding where inventions are just being made. Lisp also had to keep track of the changing IT landscape. During the 70s people were using DEC PDP computers.
Thus you find evidence for everything.
There are well-documented stable, nicely reusable, code bases in Lisp.
Clojure is most of the time married to a small eco-system: Java/JVM.
Lisp has seen and supported many more eco-systems and will see even more in the future.
> Precisely.
Enabling macros is the sole justification for the homoiconic syntax, which is often cited as the most offputting feature of lisps for uptake by large programmer teams. If you're going to have two "editions" of Clojure, one with the full feature set for language designers, and the other a more restricted sans-defmacro one for more general programmer use, then why not give that restricted one a more friendly syntax as well, or even just get them to use Java, Python, or whatever.
Macros is the way to add new special forms to a Lisp or a DSL embedded in it. This is why macros are there.
To avoud macros is to restrict oneself from designing a program as layers upon layers of DSLs which is the most powerful paradigm. Just look at that Rtml DSL of ViaWeb system.
If I want a new feature in the language is very hard to get it in and once its in its in. With macros diffrent people can do there own thing and indepently develop it.
Look for example at core.match, in every other language something like it would have been a new language feature. Clojure now has a state of the art pattern matcher without Rich or anybody doing anything.
When I say "evidence" I'm not asking for formal studies; that's too high a bar for our field. But one can at least ask to hear about specific real-world experience.
Of the two arguments, the "too-powerful" one has the disadvantage of being prima facie absurd, so I think the "large-project" one is more harmful. So, where are the large Clojure and Common Lisp projects that have been harmed by this alleged language weakness? Let's find some practitioners who actually ran into this.
For what it's worth, I haven't. Since the best thing for a large project is not to be so large in the first place, applying language constructs to make codebases smaller is a great strength when the language lets you do it—and Lisp lets you do it.
Edit: [deleted off-topic bit]
I had the pleasure of using Cascalog in production at work, which is written in Clojure. While it was in fact written by some very smart people, we had a very difficult time using some of its constructs that were very cleverly abstracted away behind macros.
The problem was that it felt nearly impossible to debug problems we had because of the long, impossible stack traces. Further, trying to get another very smart programmer to understand why some functions behaved in one way and some behaved in very different was very hard to convey. I'll reiterate that I thought the other guy I was working with was really smart, and I'm at least not an idiot, and we both felt like we had a really hard time unwrapping what the code was doing.
On the flip side, if it were written in Java (I think some parts are actually but more under the hood), you could point at the code and say "That's where the map function gets called on all the workers" (Cascalog is for Hadoop), or run the code and get some kind of stack trace where you could even begin to start figuring out what was going on. We weren't even doing anything cutting edge.
For me, I love the academic/fun endeavor. I have wasted countless hours playing and learning. But if you asked me if I would base any critical part of my production app on Clojure, especially when there are more than a couple people who weren't Lisp experts, I would have a really hard time justifying it after what I saw when I tried.
First of all Clojure has a problem with Stack traces, other lisps are much nicer in that respect (but do not have to face the JVM).
Anyway, macros can be difficult to debug, I am sure you have heard and used of the tools that usually exist in lisps, macroexpand etc. Nevertheless macros are Transformations of the AST and thus not as easily traceable as function calls.
Nevertheless, when working with ClojureScript on a web-app, I grew really fond of the possibilities macros offer, possibilities that are hardly possible with JavaScript (HTML templating within ClojureScript code, etc.). http://blog.getprismatic.com/blog/2013/1/22/the-magic-of-mac...
I wrote some macros to help with HTML5 canvas contexts and these made my code a lot more reliable and readable.
The problem with keeping languages less powerful is, that you often end up with something like Java: Surely quite understandable when you look at a few lines of code, but in the end you need a whole lot of complicated patterns and best-practices, now you get hit by a boomerang at the back of your head.
Take away message: Macros should not be used on every occassion, but they are really helpful in central places.
Still, debugging macro-using code IS harder. The first thing I need is full and partial macro expansion in the editor. There is a bit more then. When all fails I use an interpreter (most Common Lisp implementation have both an interpreter and a compiler) to follow the expansion process in detail.
If a supplied macro creates errors which are hard to understand, then it is also possible to request better compile time error reporting from the developers.
Lets be honest nobody want to write hadoop jobs directly with Java. Clojure has a query language built in that feels natural and has the full power of the language.
Other people build things like Hive or Pig that come as comply different languages.
Also known as "less is more", which is a well established point in programming, and with a lot of historical examples to showcase it.
>The other is: "it's ok, but not for large projects". I don't think I've ever seen any evidence attached to either.
Well, were are the sucesfull large projects written in Lisp (from either number of happy users or monetary success perspective)? How many are they compared to other languages?
In our field, tools get chosen not by merit but by what's the current fad. It's unfortunate, but this fact makes those two questions unhelpful in moving the discussion forward.
Since macros provide capabilities that no other feature does, it seems to me that taking them away is "less is less".
As I understand it, the intent behind "less is more" is to boil things down to a minimal number of 'things' (for lack of a better word) without sacrificing capabilities, which in practice involves getting rid of redundancies and overlap while coming up with orthogonal 'things'. Reducing the number of 'things' while also sacrificing capabilities seems to be throwing out the baby with the bathwater.
(Although to be honest, I've been unable to find a definition of "less is more" in the context of programming anywhere.)
Could you clarify what you mean by "less is more"?
Less what? Do be more precise. I've heard that with regards to complexity – not so much with regards to power.
They're a lisper who broke their teeth in Clojure. I think when people like me disdain Clojure's lispness is because we think it doesn't really teach you the philosophy behind it. This is not an argument, just my perception.
They're also in favor of code censorship (let's remember that censoring is detrimental to creative processes):
"Giving any programmer on your team the ability to arbitrarily extend the compiler can lead to a bevy of strange syntax and hard-to-debug idiosyncrasies. "
I use Racket in production along with my team and may I suggest a humble, easy solution: one person makes a pull request, another reviews the pull, and if there are new macros introduced we can discuss it with the team to see if it's necessary. It's so simple. The blogpost author is making a big deal out of nothing. To prefer a language that doesn't allow that power because the author has a problem trusting others instead of choosing to communicate with their team members is appalling.
The author also keeps mentioning Python's "simplicity". How can anything be as simple as (function args)? I'm yet to understand what people that argue this point mean by "simplicity".
Then the author talks about static checks. "How can a static analysis tool keep up with a language that’s being arbitrarily extended at runtime?". Simple, do macro expansion before static type checking, as Typed Racket does.
They're also still playing with SQL DSLs too. I think that's such a waste of effort. SQL is already a DSL for talking to the DB. I don't want another layer because I'm not going to be manipulating SQL in my code, because "SQL" has nothing to do with the problem domain I'm working on. At that point any SQL queries have already been abstracted away inside functions that have meaningful names like associate-product-to-customer or whatever. I don't want to talk about SQL ever in my problem domain abstraction layer. Using SQL DSLs as an argument against macros with the angle of static type checking is a poor argument because SQL DSLs are usually for people that use mutable code anyway. I use Typed Racket's DB library and its querying functions work together with the type system to let me know if I'm not handling some potential kind of value that might come from the database.
The author then mentions Unix's consistency. Unix couldn't even decide on a standard notation for command line arguments. Then onto that fallacy that reality is object-oriented. Objects can't possibly be as composable as functions because Objects break down into methods (which are not composable, are not first class, etc) whereas functions (lambdas) can make up everything and can really be thought of as an atom for computation (i.e. Lambda Calculus).
Complaints like "Clojure has nine different ways to define a symbol" are moot. Pick one that your team likes and go with it. On to the next thing. Also, to argue against Lisps by arguing against Clojure is like arguing against democracy by arguing against the Democratic Republic of the Congo.
I do believe the blogpost author is severely misguided in their criticism. To say things like "Python wants the conceptual machinery for accomplishing a certain thing within the language to be obvious and singular" while ignoring the fact that lisps machinery is obviously much simpler and obvious and singular - again, (function args) - is disingenuous. It does make me believe that all their SICP reading was for nothing (I've only lightly skimmed SICP and I don't pretend to have read it).
The author does acknowledge (en passant) that certain Schemes (they don't identify which) don't suffer from this complexity (which makes the whole post look more like a criticism of Clojure). I'd invite the author to look into Racket.
They say that lisps "impose significant costs in terms of programmer comprehension". My experience is that if you divide your layers of abstraction correctly you will be able to work in the problem domain layer where nothing is obscure. And that layer is built from smaller in the layer below, parts that are also clear in what they accomplish because they only do one thing in their abstraction level. I've found that following this rule of only doing one thing per function makes for code that is easy to understand all the way from the bottom to the top layers. Programming this way, however, is the classic, boring way to write code [1], and because it's not a fad I guess people aren't too much into it.
Also to the point above, having already rewritten a significant portion of a Rails legacy app into Racket with the help of my coworkers, it seems that lisps introduce more understanding and shed more light onto the code, precisely because it makes everything explicit (we code in functional style so we pass every argument a function needs) and does away with "Magic" that Rails and rails fans like so much. When something gets annoying to write we implement our own "magic" on top of it, not in terms of silly runtime transformations that lesser languages like Ruby need to resort to, but through dynamic variables (Racket calls them parameters), monadic contexts, etc, i.e. things that can be checked at compile time.
And finally: "I think it’d be irresponsible to choose Lisp for a large-scale project given the risks it introduces". Well, the only risk I've personally witnessed is the very real risk of your coworkers starting do dislike more mainstream, faddish languages like Python and Ruby, because they don't allow the same freedom and simplicity and explicitness that lisp does (lisp has a long tradition of making things first-class, which consequently makes these things explicit).
[1] We use top-down design to decide what the interface for a given abstraction layer will look like, and bottom-up to decide which functions should be written in the layer below; then we cycle that process by refining the layer below through the same process of defining its interface top-down and then the layer below it as bottom-up. And we use algebraic type checking along with contracts to enforce post-conditions and properly assign the blame to the right portion of the code to speed up debugging. These are all old techniques.
Intuitive = FamiliarIt's a valid complaint. I think the quote by John Carmack: "Everything that compiler will allow will at some point be written" Describes the situation well.
In other words, since Lisp pretty much allows all, anything will be written in Lisp. When you look at Lisp function, it might do what you want, or it might not. There are no guarantees it does what it claims to do. So, chances are you either have supreme belief the guy behind it did his end of the deal, or you are writing your own. And with all the power Lisp offers why not write your own? And this is generally what I think is the failing point. It brings NIH of the highest magnitude to software development.
Tests, contracts, type checking - those are the guarantees. It's not unlike any other language.
http://www.chiark.greenend.org.uk/~pcorbett/yvfc.html
Which, as you can immediately tell, is a Lisp interpreter written in Python, as a single expression. No macros, though.
(Edit: typo)
I don't really understand that point. In Racket, for example, programs macro-expand down to a very small set of primitives, such as `let-values` and `lambda`. This makes it easier to do analysis, not harder. For example, this is how something like Typed Racket can support every idiom in untyped Racket programs -- because they all expand down to a fairly manageable core set of forms. (Or if you need to analyze something no-so-primtive, you can stop expansion on whatever that is.)
Racket is descended from Scheme. I don't know if CL or Clojure expand down to quite such a small primitive core, but I imagine the story is roughly similar?
Anyway, writing such tools is not what the average programmer would do on a putative large project.
Any large project needs technical and social norms, mentoring, and leadership -- regardless of language. I think the language is the smallest part of it. Perhaps like how in security it's social not technical engineering that usually turns out to be the weakest link.
The fact is that Clojure/etc is much easier to analyze statically than the popular everything-is-a-dictionary scripting languages. That should be obvious given that the reference implementation is a compiler, not an interpreter. But stuff like Typed Racket and Typed Clojure should eliminate any remaining doubt.
I don't think the author understands that these things happen at different times. Especially because they said "at runtime" when macros expand much before that.
On the contrary, something like Rails is what I'd call extending the language arbitrarily at runtime. So the very lack of macros is what motivates shaky transformations at runtime.
This points out some very real potential dangers of large-scale collaboration with Clojure (and presumably some or many other Lisps/Lisp-likes).
However, I think the conclusion is overstated. Yes, based on what's provided, it may take more discipline and better, more explicit processes for a team to effectively collaborate in Clojure than in Python (using the author's running comparison). Yes, if we take this at face value, it does appear that people who depend on static analysis might find Lisp lacking.
But does this tell us why Lisp isn't widely used in industry? If we assume that "widely" means "as widely as Java or Python," which seems to be the statement made here, I don't think it's valid to cite the provided complaints as most or even a large portion of the explanation. The fact that there are no or almost no mainstream educational institutions teaching new students Lisp seems to me a far more likely candidate for front-runner on this issue.
That it shouldn't be widely used is a little easier for me to agree with, only because I'm on board with some of the points here about the discipline and extra work it would take for a large team to effectively cooperate given the malleability of Lisp. I've worked with enough other programmers to know that kind of care and attention to process are very rare (and this isn't "all of you suck;" I know I have and will again cut corners and ignore protocol in situations where time or resources make it hard to do things perfectly every time).
Also, I think the author's last point is important to mention, because it'd be easy to miss it: He's not arguing that Lisp sucks. He states explicitly that it's great in at least some ways. I just don't think the black and white claims being made about its practicality are quite supported.
I don't know about that. Good code is good code. It's sort of one of those, "I'll know it when I see it" things.
The ease of which you can code classes for the sake of classes in Python can make some really hairy code out of what should be simple programs. Was that necessarily the 'one right way to do it'? Who's to say. And all the static analysis tools and syntactic aren't going to undo those hairballs anytime soon.
You might just feel more comfortable with languages with lower code density. 'brandonbloom made a good blog about that[1]. I think it can be doubly applied to any situation where meta-programming is employed.
1: http://www.brandonbloom.name/blog/2013/06/24/code-density/
Granted, space leak issues are pretty difficult to analyse, so it makes the language seem hard to use in practice, but that's because all the low-hanging fruit like type errors are solved by how the language is designed, so you only end up with the hard bugs.
This is not correct. To understand why, please see Ryan Culpepper's answer to this SO question:
http://stackoverflow.com/questions/7046950/lazy-evaluation-v...
They aren't that many to begin with, so it could just be (self-)selection bias.
Forth people do fine using Forth too, but I don't see that as a point that it's an appropriate language for most projects and/or people.
Have we not learn by now that these systems are not easy to reason about. Are not all the things one first learns (ie Animal -> Dog) bullshit and should be avoided.
Why is it in every good OO book that, composition is better then inheritance. Why is every OO book full of examples about how to avoid mutabiltiy and make the system easy to reason about?
The idea that OOP systems (as generally) thougth of goes completly out of the window as soon as you have any kind of concurency, even just event handling.
> which rejects OOP
It does not reject, it takes the usful features like polymorpism and gives them to you. Protocols are better then interfaces, better then duck typing.
> In Clojure, if I want to define a symbol there are nine different ways of doing so.
There are a lot more then nine. But I would recomend rich or stus talks on simple vs easy. Just saying there is nine of something and thus its complicated is idiotic.
Java has only one thing, classes, does that make it simply, or does that just mean that its hoplessly overloaded?
Clojure is extreamly simply. State can only live in a var, atom, ref or agent. Every one of these has clear semantics, this includes clear sematnics in a multithreaded world. No other language has such clearly defined state management.
> Clojure claims to include these language features as a way to mitigate the complexity of parallelism; frankly, I’ve never found threading or interprocess communication to be any sort of conceptual bottleneck while working on some fairly complex distributed systems in Python.
Distributed system != Shared Memory
Nobody, really nobody can say taht distributed systems are easy. Just listen to the people that implment this stuff. But it is clear that a language generally does not really help you with reasoning about that system.
However when you run on a 16 core with shared memory and you have to do lock ordering and all this stuff,then you will defently be happy for the tools that clojure provides.
> Less is more (as long as “less” is sufficiently convenient).
Clojure is actually a much smaller and much simpler langauge then python every can hope to be. Clojure is simple, and strives for simplicity in every feature of the langauge. See here:
- Simplicity Ain't Easy - Stuart Halloway http://www.youtube.com/watch?v=cidchWg74Y4
- Simple Made Easy http://www.infoq.com/presentations/Simple-Made-Easy
Article simply says that giving ALL programmers power to design language leads to bad things. Lisp, Clojure, etc. And I can see why. People love making their own languages, it's fun, but a good programmer and a good language designer are two mostly unrelated things. Good programmer often needs to look at problem from a weird angle, while a language designer needs to find shared views. I'm not saying they don't have a lot in common as well, but I can see how programmers can design AWFUL languages.
Note: Good programmer means a good general programmer i.e. someone that solves various tasks in his favorite programmer language.
1. > But, that doesn't mean that modelling hierarchies is not necessary in some domains.
Agree but the addition of full OOP seams overkill to reach this goal. Look at this clojure code:
>(derive ::rect ::shape) >(derive ::square ::rect) > (parents ::rect) -> #{:user/shape} (ancestors ::square) -> #{:user/rect :user/shape} (descendants ::shape) -> #{:user/rect :user/square}
Clojure gives you hierarchy 'À la carte'. This means that you know longer tie the two things together, it easy in clojure for example to have many diffrent hierarchy that are independent but still dont get in each others way. Modeling the same with objects is hard. Just a example, for often good reasons multiple inheritance is not allowed in most languages, however if you use hierarchy as a domain model and not as programming model you generally want it.
2.
I agree with the articles point, people should not invent there own langauges for everything, however that is a terrible reason to discard the language for 'large scale' production use. Every language has features that generally should be avoided, every language make it easy to do the wrong thing. Macros are relatively easy to understand, compared some other language features I could name. Also the effect of macros is generally local, unlike say monkey patching.
Note that this refers to the Nygaard interpretation of OOP, which is also the most widely used: rigorously class-based and in many ways retaining a procedural nature.
Smalltalk and Eiffel are different beasts, but they never really made it.
C has all these problems as do C++, Java, Python, Ruby.
On the contrary, beauty of Lisp comes from being accidentally discovered minimal set of unique interweaving features (very few special forms, high order procedures, list structure, to glue code and data together, macros, the way to use Lisp as a meta-language to itself, type-tagging of values, instead of declaring variables, and, the numeric tower, and lexical scooping, as recent addirion).
This set of features is very balanced and good-enough. Adding more "foreign" features actually ruins the balance.
Having "just this" and following the paradigm of layered DSLs embedded in a Lisp, popularized by SICP, lots of complicated things could be created, including CLOS which is nothing but a DSL (a bunch of macros and procedures).
This notion of what a Lisp is actually helpful to understand not just Clojure but also Haskell, which is, in some sense, also a small kernel based on Lambda Calculus and tons of syntactic sugar.
By adding more and more features without breaking the balance we got Common Lisp, while an attempt to keep the balance produced R5RS.
If we switch the perspective on what Clojure is to the notion of a scripting language for the JVM (a-la Ruby) with sometimes looks like a Lisp and sometimes behaves like a Lisp, but strictly speaking not a Lisp, because the unique balance was ruined by adding stuff, everything, it seems, falls into its places.
Arc is a dialect of Lisp, Clojure is language of its own, but marketed as a "modern dialect of Lisp", which makes no sense.
So to make static analysis tools useful on macros, they really need some way to map back to the original source forms. Not all tools do this (either at all, or well) - and to the extent that they don't it is an big impediment for static analysis.
Also the killer challenge: macro expansion in Clojure can depend on a mutable environment at the time of macro expansion. This makes it impossible to do reliable static analysis, unless you are able to recreate the runtime environment at the time of macro expansion in your static analysis tool, which is hard/impossible in general.
This is part of the motivation for my little Kiss language experiment: with immutable environments you can keep the power of macros, but avoid the mutable environment problem. Ideally, macro expansion would be governed only by things that are provably compile-time constants (not sure how feasible this is while maintaining the dynamic flexibility of Clojure that we all love... but it's an attractive idea at least).
Learning a language involves more than just learning syntax and semantics. You also need to learn how to write for maintainability. It sounds like the author is less certain about how to do that with Lisp, but instead of seeing it as a chance to learn more he writes off the entire Lisp family as impractical.
I guess there are several ways to respond at that point: one is to persevere, learning more and coming to a better understanding of what advantages and disadvantages are, as well as the appropriate use cases.
Another is to just give up and write a proscriptive blog post, possibly also with a biased, incomplete comparison of the new thing to an old thing.
The latter approach is extremely frustrating for several reasons: 1) often, the authors ignore or fail to grasp both the pros of the new thing as well as the cons of the old; 2) the proposed solution is to throw out the baby with the bathwater, instead of to figure out how to improve the new thing; 3) it provides fuel for others' confirmation bias.
How large is the scheduler for the Hubble Space Telescope, which has been adopted for many other telescopes?
Who are the users of AllegroGraph?
How complex is PTC's CAD system which uses Common Lisp? A few years ago they mentioned 7 million lines of Lisp.
The author cites Korma as an example of a library that uses macros, but Korma only uses macros to provide a small degree of syntax sugar. So instead of writing:
(exec (where* (select* people) `(> :age 18)))
(select people (where (> :age 18)))
There are some libraries, such as core.async or core.logic, that make more extensive use of macros, but these libraries are relatively rare, and take a lot of work to get right. It's something I'd expect to see in a dedicated library, not as part of a solution in a large software project.
Worrying about overuse of macros in Clojure is a bit like worrying about overuse of FFIs in Python. Sure, it's possible to abuse in theory, but it's not really a problem in practise.
1. I'm able to concisely express what I'm trying to do.
2. I never, never have to copy/paste or otherwise do repetitive work because one case is slightly different than another.
Number 2 is probably more important to me. It's why I'm happy when a language has first class functions and closures, and why I'm unhappy when the language is Java.
I think the cases where you can't modularize things enough such that you have to essentially include two version of the same thing is a programming language expressiveness failure.
Finally, I strongly disagree that programmers aren't or shouldn't be language designers. Nearly every worthwhile program I've written that people have used has needed an API or scripting/query language. So any language had damn well be suited to writing a parser. LISP fills that bill too.
If LISP has shortcomings, its expressiveness and power are not them. If you want to complain about a lack of consistent implementations, lack of libraries, difficulty interoperating with any other language, lack of a canonical free implementation, no decent supported GUI toolkit, or the impossibility of distributing binaries real people can use without spending thousand of dollars on a professional LISP environment, go right ahead.
"Lisp isn't a language, it's a building material" - Alan Kay.
Lisp is an opportunity to build a DSL that fits any given problem domain elegantly.
In the end, they only directly added support for 1) type-safe macros, aka "black-box" macros 2) ...invoked transparently as methods, aka "def macros"
They identified "white-box" (non-type safe) macros and quasiquoting as distinct from black-box macros, because the type signature of a black-box macro tells you approximately what it will do, meaning that you can treat it as a "black box". But additionally, in order to write a meaningful type signature, the input to a black-box macro must _already_ be valid scala code! This means that the addition of macros cannot actually result in new scala syntax [d].
[a] https://news.ycombinator.com/item?id=3709193
[b] http://www.infoq.com/presentations/scala-macros
[c] still hoping the existence of scala macros leads to the deprecation of a bunch of other features though
[d] see the explanation starting around 33m30s in [b]
I disagree with this. Ask any non-programmer "is a square a rectangle?" Or "is a list of triangles a list of shapes?" and they will say "Yes!"
Yet as soon as either of these becomes mutable it all falls apart and intuition fails. You cannot change the height of a square independently of its width, you cannot add a square to a list of triangles (but you can add a square to a list of shapes).
Human beings reason "immutably".
"Fine only do OO with immutable objects!" I hear you say. It's a good idea buy you're now on the path (that I took a few years ago) towards functional programming.
This is why I think that OO, and more particularly mutable state, are quite difficult for new programmers to grasp.
"OOP is widely-used and easily comprehended because it is a fairly simple way of modeling reality that is compatible with how human beings do it."
Close, but no pants, Buckwheat!
I don't code much lisp these days (though getting into clojurescript a bit now), but when I used to, the cycle went pretty much like this -
1. Express what you want to express as an s-expression, capturing known structures in the simplest way I can think of.
2. Figure out which aspects can be "functions" straight forwardly and which are macros and implement them.
3. Test and iterate a bit till I like the way domain elements are composing and the way the composition looks in code. Try to reduce the required concepts in each iteration.
4. Document the relationships that have emerged from this process so others can understand it.
5. Usually I'm done, but sometimes (the few) users of my "api" come back with questions, based on which I iterate a bit more.
I've mostly followed this in building the "editing style specification language"[1] part of the product "muvee Reveal" (an automatic video editor)[2], built in a custom scheme dialect called "muSE"[3]. (Full disclosure: I've happily worked for muvee Technologies from 2002 to 2011.)
Btw - most discussion on macros and lisp seem to first assume that there two things - a) functions and b) macros. There are more, depending on the kind of lisp system you're working with.
You could form a taxonomy of sorts based on whether argument forms are evaluated and whether the result form is evaluated -
1. Argument forms are evaluated before "apply", result is a value (i.e. not evaluated). => Function
2. Argument forms are unevaluated before "apply", result is code (i.e. is evaluated). => Traditional macro
3. Argument forms are evaluated before "apply", result is code (i.e. is evaluated).
4. Argument forms are unevaluated before "apply", result is a value. => Traditional macro (depending on system)
In the course of using the domain modeling approach above, I've written stuff like functions that create one or more macros, macros that evaluate to function values (not s-expressions) and such stuff that might be considered an "abomination" by the OP ... but in the context of the domain, the concepts are usually clear enough to be used without major issues.
[1]: https://code.google.com/p/muvee-style-authoring/ [2]: http://www.muvee.com/en/products/reveal [3]: https://code.google.com/p/muvee-symbolic-expressions/