* poor ecosystem of libraries - few gems, most other half-baked
* poor community coordination
* Dependency management limitations with quicklisp
And some specific red flags like:
* poor support for json[6] * poor support for async
* have to restart the server every 20 days because of some memory leak [3]
* hack to tune GC [5]
If you are using lisp in production for non-trivial cases, do these issues still exist? is there a way you can quantify effort is resolving them, and if yes, what is it? and, finally, if you had to re-do your project, would you chose lisp or something else?
[1]: http://www.gigamonkeys.com/book/
[2]: https://leanpub.com/lovinglisp/read#quicklisp
[3]: https://lisp-journey.gitlab.io/blog/why-turtl-switched-from-...
[4]: https://lisp-journey.gitlab.io/blog/why-deftask-chose-common...
[5]: https://www.grammarly.com/blog/engineering/running-lisp-in-p...
[6]: https://stevelosh.com/blog/2018/08/a-road-to-common-lisp/#s4...
The historical timeline is especially interesting because Reddit's cofounders Steve Huffman & Aaron Swartz were alumni of Paul Graham's first YC batch and PG is the author of the well-known Lisp essay "Beating the Averages":
- 2001-04 Paul Graham : Lisp essay "Beating the Averages" [1]
- 2005-07-26 Paul Graham : https://groups.google.com/forum/#!topic/comp.lang.lisp/vJmLV...
- 2005-12-05 Steve Huffman : https://redditblog.com/2005/12/05/on-lisp/
- 2005-12-06 Aaron Schwartz : http://www.aaronsw.com/weblog/rewritingreddit
The takeaway from the Reddit case study is this: Yes Lisp macros and language malleability gives it superpowers that other languages don't have (the "Blub Paradox") -- but other languages also have other superpowers (ecosystem/libraries) that can cancel out the power of Lisp macros.
You have to analyze your personal project to predict if the ecosystem matters more than Lisp's elegant syntax powers.
Once they needed to grow by adding developers, they decided to rewrite into a language that other people already knew.
The only "Super Power" is the complete understanding of what your endusers want so that you can grow.
In production using SBCL on a customer project we had good results hiring a SBCL maintainer/developer to work past a few issues. From personal experience, I can assure you that LispWorks and Franz both provide incredibly good support for their customers.
That said, if I had to develop Reddit right now, I would not use Common Lisp either.
Also, quicklisp didn’t exist/wasn’t as widely used then: I’ve done all sorts of Lisp projects now and never had serious issues finding good libraries for standard web development tasks.
Clojure?
Or a non-lisp language instead?
Indeed. The "blub paradox" is profoundly arrogant - it presumes not only that people are too stupid to see the advantages of Lisp, but that Lisp advocates are smart enough to know that other languages have nothing unique to offer that Lisp doesn't.
You are right, though, that the Lisp community read the essay and internalized that all other languages are Blub. In my opinion this led them to collectively dismiss, for far too long, Dictionary Literals in every dynamic language which has eaten Lisp's lunch in the marketplace in the last two decades. Objections were raised ("you need homoiconicity for macros!") and everyone assumed they were valid and stuck with Common Lisp's woeful hash table interface and the loop macro keywords, insisting all the while "oh, that's a DSL, you could implement that with Macros"[0].
And then Clojure implemented Map literals and - what do you know - Clojure still has Lisp Macros!
[0] Efforts in this direction thankfully forthcoming: https://github.com/inaimathi/clj
I don't think that the fundamental flaw with that essay is the arrogance so much as the assumption that "language power" lies on a continuum in the first place.
That original Reddit system was very simple, yet the Lisp source code was of poor quality (it is currently available on the web). When it was released, the lispers at comp.lang.lisp said they could write a clone in a few days. One of them did it in about 2 or 3 hours (true story).
So it wasn't good code. Moreover they, for some strange reason, chose to develop using a different implementation than the one in the server, and things like that.
Then Reddit got a famous name recommended by PG, who was a Python expert. So he rewrote the thing in Python and obviously, being a Python expert, did a good job.
There's a perceptual litmus test here. Some people see life on earth and see God. Others see the faintest exploration of the design space, an impoverished code base that made it impossibly far. A miracle, perhaps, but not what we'd see in a billion runs of the simulation.
Same with Lisp macros. Their few variants look bolted on. We haven't begun to explore the design space, yet the best macro systems in other languages are copies of what lisp has figured out so far.
There's an 80:20 rule for language emergence. Ecosystem questions aside, a Turing-complete language needs to start by getting one thing right. APL leveraged multidimensional array handling into a general purpose language. Scripting languages gain much of their leverage from nested hash tables.
Concise versions of monadic parsing are exhilarating in their power and expressiveness. If a language was designed so parsing and manipulating itself was its sweet spot, generalizing this parsing to typed trees would be easy, and we'd invent many new programming paradigms. A general purpose language would follow.
Lisp, an implementation of the lambda calculus, is not this language. Lisp is a machine language precursor to this idea, with various macro systems bolted on.
Haskell, an implementation of typed category theory, is not this language either. At the expression level, Haskell is stuck at a few hard-wired pattern matching primitives. Haskell does not make the assumption that everyone has learned to compose arbitrary parsing operators.
Common Lisp is generally the first thing I think of for any new work I undertake. There are exceptions, but usually my first choice is Common Lisp. It has been for a little over thirty years.
I've often worked in other languages--usually because someone wants to pay me to do so. I usually pick Common Lisp, though, when the choice of tools is up to me, unless some specific requirement dictates otherwise.
The objections you list might be an issue to some extent, but not much of one. Certainly not enough to discourage me from using Common Lisp in a pretty wide variety of projects. I've used it for native desktop apps, for web apps, for system programming, for text editors, for interpreters and compilers. When I worked on an experimental OS for Apple, the compiler and runtime system I used were built in Common Lisp.
I'll use something else when someone pays me enough to do it. I'll use something else because it's clearly the best fit for some specific set of requirements. I'll use it when there isn't a suitable Common Lisp implementation for what I want to do. I'll even use something else just because I want to get to know it better.
I've taken various detours along the way into learning and using various other languages. I like several of them quite a bit. I just don't like them as much as Common Lisp.
The pleasure I take in my work is a significant factor in my productivity. Choosing tools that offer me less joy is a cost I prefer not to bear without good reason. That cost often exceeds the advantage I might realize from using some other language. Not always; but often.
There was once a language I liked even better than Common Lisp. Apple initially called it 'Ralph'. It evolved into Dylan, which, in its present form, I don't like as much as Common Lisp. If I or someone else invested the considerable time and effort needed to write a modern version of Ralph, then I might choose it over Common Lisp.
For now, though, I'll stick with the old favorite. It continues to deliver the goods.
The surface language was basically Scheme, with some special forms removed and others added or renamed. The type system was basically a simplified Common Lisp Object System; all values were instances of CLOS classes.
The compiler, runtime, and development environment were built on Macintosh Common Lisp. Ralph inherited MCL's design and aesthetics.
Like Common Lisp, it was designed for writing software by interacting with it and modifying it as it ran. That's the essential point.
The project I worked on, bauhaus, was running basically all the time we worked on it. We worked on it by interrogating and hot-modifying the running system, in the time-honored tradition of old-fashioned Lisp and Smalltalk systems.
Later Dylan was not like that. It evolved into just another batch-compiled application language. You edit some files. You batch-compile them. You run the resulting artifact for testing. Rinse and repeat.
That's how OpenDylan is today. That's why it lost me.
It's also what's wrong with some newer Lisps, from my point of view. They aren't designed for building programs by modifying them from inside while they run. They don't know how to handle existing data in the presence of redefined types. They don't provide interactive debuggers and inspectors that can be used to rummage through the memory of the running program, edit it while it runs, and conitnue execution.
Common Lisp provides these features, and has done for decades. Ralph provided them. Smalltalk does, too. If all the Common Lisp implementations in the world suddenly evaporated, I could get along fine with Smalltalk, after a brief period of adjustment. It's probably purely an accident of history that I'm a Lisper rather than a Smalltalker, anyway.
But Dylan doesn't have those essential features anymore, and nothing that lacks them is likely to win me over--not OpenDylan, not Julia, not Clojure or Haskell, or F#. I like all of those languages, but not nearly as much as I like my livecoding.
Ralph used prefix notation, dylan infix notation. Here you can find an implementation of ralph:(2)
From (2): Ralph is a Lisp-1 dialect that compiles to JavaScript. It is heavily inspired by an early version of Dylan (also known as "Prefix Dylan"), as described in Dylan – An object-oriented dynamic language.
Also (3) is about a Lisp to Dylan translator. There is also norvig lisp to dylan translator cited in (3).
(1) https://en.wikipedia.org/wiki/History_of_the_Dylan_programmi...
(2) https://github.com/turbolent/ralph
(3) https://tim.pritlove.org/2003/11/10/converting-lisp-to-dylan...
Off topic, but many years ago I had lunch with Larry Tesler (John Koza was also there) and Larry knew of my CL book from the 1980s and pitched me to rewrite it in Dylan. That might have changed my career trajectory.
I liked and admired Larry. I considered him a friend. Once I sat with Steve Jobs in my office at NeXT and argued about Larry's virtues. I took the pro-Larry position.
Your lunch was probably around the time I was working on that Lisp OS at Apple. That was Larry's idea, too.
If you don't hear from me, it's not because I don't want to talk with you; it's because I'm forgetful (one reason I like email better than synchronous calls. Inboxes remember things that I forget.)
Knowledge Graph Navigator sort of intersects with some of my running interests--particularly knowledge representation and frame stores.
https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...
To be expected, I suppose. It really was my favorite working language ever.
I'd probably use SBCL or CCL for everything, if Lispwork' CAPI or something equivalent worked on them. I do use them when I don't need to build a GUI that I can easily port among platforms.
When I wrote a WYSIWIG text editor years ago, I did it with CCL because it was for my own use, and I didn't need it to run on platforms other than macOS. If I wrote a new version of it, I'd try to find a way to make it work across Windows and Linux, as well. (I can't use Lispworks for that because it's violates their license restrictions on distributing something that can be used as a Lisp development environment.)
I use other languages when I'm paid to, or when I want to do something that is a lot easier to do in another language. For example, I recently wrote a prototype of a web-based collaborative editor and publishing platform for shared documents. I chose Couchbase for storage, because it solves the sync problem reasonably well, and I used Clojure and Clojurescript with React and Reframe, because it worked well with the Couchbase Java SDK.
I've been paid to build prototypes, apps, tools, and system software with Lisp, Icon, ML, C, C++, Objective-C, Java, Python, Swift, Prolog, Haskell, Scheme, and other things. If I had a modern version of Ralph, I'd most likely use that. Failing that, Common Lisp is what I like best.
TO address some of your points;
poor ecosystem of libraries: many great libraries, some incomplete. you must not be afraid to look at the source code to understand and edit
poor community coordination: true, there are few large scale community lisp projects. be the change you want to see
dependency limitations with quicklisp: like what? quicklisp works great. if you are talking about loading several versions of the same library at the same time in a single image, this is being researched and developed
poor support for json: no, there is good support
have to restart the server every 20 days: not familiar, I have had lisp servers running for years now
hack to tune GC: no
if I had to redo my project would I choose Lisp? I would choose lisp, yes. there is no more powerful language than this, and believe me, I have tried many
> For me, the most important quality
I need in a JSON library is an unambiguous,
one-to-one mapping of types. For example: some libraries
will deserialize JSON arrays as Lisp lists,
and JSON true/false as t/nil. But this means [] and false
both deserialize to nil, so you can't reliably round trip anything!
[1] https://stevelosh.com/blog/2018/08/a-road-to-common-lisp/#s4...
USER> (yason:parse "{\"null\" : null, \"false\" : false, \"empty\": []}"
:object-as :plist
:json-arrays-as-vectors t
:json-booleans-as-symbols t
:json-nulls-as-keyword t)
("null" :NULL "false" YASON:FALSE "empty" #())
All those options defaults to special (dynamically scoped) variables, so you don't have to pass them everytime.
Encoding prints to the standard output (you can redirect it to an http stream, if you want), so you don't see double-quotes being escaped, but the string is equivalent (modulo spaces):
USER> (yason:encode-plist *)
{"null":null,"false":false,"empty":[]}
That's his opinion. We have a lot of JSON libs, maybe too many. I use "jonathan" and i'm happy with it.
EDIT: He wrote: "JSON support in Common Lisp is a god damn mess. There are an absurd number of JSON libraries and I don't really like any of them."
He doesn't like any of them, but I do like more than one.
With regards to Steve's issues: ask yourself whether they apply to you.
If you want a full blown Common Lisp experience, with a GUI based REPL, GUI debugger with edit-and-continue, and all the nice stuff you see from Xerox PARC, TI and Genera demos, they are the ones to get.
The developer experience was great. I get subsecond hot-reload on all plataforms (app emulator/app on-device/web/server).
The same UI components was shared between ReactNative and ReactDOM. All App issues are reproducible in Web, then I fix in web and I'm sure that App is working.
Even SSR was possible, with some tweaks.
Barring that, as a sibling comment asks, what specific framework and/or libraries did you use to achieve the holy grail of sharing-all-the-things? (thought ReactNative required implementing separate UI for iOS and Android).
Sub second hot reload sounds amazing, btw.
For builds and dependency management: Leiningen, boot or deps.edn
For the front-end (clojurescript): Shadow-cljs + reagent
For the backend (clojure): Nrepl and nrepl.cmdline, Ring, jetty + compojure for a REST API
Frontend and backend can share code with .cljc files that work on all clojure dialects. https://clojure.org/guides/reader_conditionals
Both nrepl and shadow-cljs support code reloading through the REPL connection.
You can compile to a jar file and deploy that, build a docker image, or compile to a native executable with graalvm, or deploy on Aws lambda with Cljs+Lambada. Lots of options.
We have used Common Lisp in production at Rigetti Computing for 4 years both in cloud deployment as well as for a downloadable SDK. Common Lisp is used to build what is currently the best free optimizing compiler for quantum computing [1, 2] as well as one of the most flexible quantum computer simulators [3]. They’re both open source, so you can actually see for yourself how we version, deploy as binaries, and deploy in Docker containers.
> If you are using lisp in production for non-trivial cases, do these issues still exist?
Before getting into the specifics, any large, non-trivial product that customers use will need tweaking. Especially languages that have runtimes, including languages like C. This is usually a result of leaky abstractions that just have to be dealt with in The Real World (TM).
> * poor ecosystem of libraries - few gems, most other half-baked
You have libraries for most things you need. Some are lacking, like in scientific computing. But all your usual stuff to glue things together is there and works.
Some people confuse “half-baked” with “no commits for 5 years”. Lisp libraries are older than many in vogue languages, and Lisp doesn’t change, so “5 years no commits” shouldn’t be worrisome.
At Rigetti, the biggest weight we had to pull that we wouldn’t have needed to do in, say, Python, is bind to a bunch of FORTRAN code, specifically LAPACK. That was a pain because you need to do FFI with complex double floats.
* poor community coordination
The community all agrees on one thing: Lisp is standardized.
Lisp lacks a sort of “centralized community” like Go or Clojure. I like to think that Common Lisp is more “democratic”. We have a “constitution” (the Common Lisp standard) that we all agree on and we all make our own contributions atop that.
With that said, #lisp on Freenode is attentive. SBCL, for example, responds to bug reports almost immediately. Several library ecosystems are kept up to date, etc.
> * Dependency management limitations with quicklisp
With Quicklisp, I personally realized a lot of other dependency management solutions are overblown and don’t actually solve problems. Maybe QL gets away with it because the ecosystem is smaller than others.
Xach, the creator of QL, does a lot of community diligence for free to ensure things work.
For production code, I’ve had 0 issues.
> * poor support for json
We use YASON [4] and it works fine.
> * poor support for async
This is true now but wasn’t true years ago. Lisp implementers have not been interested in implementing a runtime supporting async programming. Little in the Common Lisp language itself prohibits it. Before the world of threads and SMP, Lisp implementers did implement async runtimes.
Why aren’t Lisp implementers interested in async? I’m not sure. Maybe they see it as a passing fad?
> * have to restart the server every 20 days because of some memory leak
Memory leaks can happen but they’re usually the programmer’s fault for not understanding symbol interning, weak pointers, or finalizers. Writing long-running applications does require know-how.
> * hack to tune GC
I don’t consider it hacking. In SBCL we tune the GC to our application. This seems reasonable; different programs have different allocation patterns. But you wouldn’t need to do this out of the box.
> is there a way you can quantify effort is resolving them, and if yes, what is it?
Big applications that are customer products require care and feeding. The benefits of writing in Lisp in the first place largely outweigh some of the extra work needed to productionize it.
I will say that writing production Common Lisp code is greatly benefited by having at least one experienced Lisper on the team. If a team would, for some reason, write an application in Lisp but not really have a good understanding of the language, then productionizing and hardening will be difficult.
> and, finally, if you had to re-do your project, would you chose lisp or something else?
Common Lisp allows something that never existed before (a completely automatic, optimizing quantum compiler) to exist. If Lisp weren’t there, that wouldn’t exist.
My parting advice is this: The biggest difficulty in writing an application is to actually write it. It takes a lot of hard work to get requirements, listen to feedback, articulate a direction, etc. In Common Lisp, SBCL or LispWorks or whatever else, dumping an executable is so easy and you can have a deployable “production app” in 5 minutes. So the good news is that you won’t be held up trying to figure out how to dockerize your Python app.
I definitely agree that before embarking, doing an objective and relevant comparative analysis would be good (we did this at Rigetti), but ultimately you just need to sit down and write.
Almost every programming problem is solvable these days with enough glue and elbow grease. I wouldn’t be too worried.
[1] https://arxiv.org/abs/2003.13961
[2] https://github.com/rigetti/quilc
The library: https://github.com/rigetti/magicl
Comparison with NumPy: https://github.com/rigetti/magicl/blob/master/doc/high-level...
XML is totally fine for storing structured dumb data. Don't program in XML.
I would not be so bullish about implementing your own interpreter - as the best obvious choice every time.
Numbers especially are hard to get correct unless you know what you are doing.
The point is complex configurations tend to grow into a poor programming language. They would be far better as a lisp DSL.
So I think GP is correct.
I experienced this in both Clojure and Ruby as well. The fact that developers can spend countless of precious time, cycling through a loop of malleability, reforming an already functional product in production. It costs precious time and money, it is expensive. And more importantly, it is non-stop.
Other programming languages like Go are very conservative with feature changes, and imposes hard limitations by choice, the team/company behind it thinks the same way, that a cycle of O(n) is destructive.
And nowadays, you can't expect a developer to stay on a company for so long, so a sureball tooling to hit it hard, and precise, with less time and tinkering around is much needed than ever.
On the future of programming languages, we will see languages being used not because it is malleable, but because it is sure, precise and right for the job.
- I have to use every variable I declare, otherwise it won't compile (b/c golang authors decided this produces better code)
- Official docs confusingly suggest I structure imported files with the root as github.com/ (b/c golang assumes all code will be open sourced)
- I can only import one file per folder
- There is no built in map or reduce function. You're expected to use for loops everywhere because it is more explicit
Once I discovered the last point I noped out.
> - There is no built in map or reduce function. You're expected to use for loops everywhere because it is more explicit
Go is an imperative language, where no implicit magic happens on the background, like auto creation of resources on runtime, every sequence needs to be manually defined by the programmer. "With imperative code, you’re telling your program how to do something. And with declarative code, you’re telling your program what you want to do. - Tyler Mcginnis, React Fundamentals", depends on the programmer, but I like to be in control.
The Clojure language and core library is extremely stable, almost uncannily so. I can't think of any major upheavals it's had for a long time. Transducers from 1.7 maybe? But those fit in with the rest of the core library and didn't cause any issues.
With Lisp, you can create a "Hello World" program in variety of multiple ways. Sure you can refactor it, starting with how to print the letter "H", throw UTF-16 support there as well. At the end of the day, you wasted precious time and resources printing "Hello World".
Doing things in variety does not mean value.
Doing it right the first time means value, now imagine if it's a team.
If you need a low memory consumption then Clojure is also not a great choice.
We've built an ecommerce site on Clojure though and it's been great. :)
The only problems we are seeing is the significant startup time cost (hence the Go bits, where it matters).
Memory usage also isn't always great, but that's more a JVM problem, as it's not releasing reclaimed memory as quickly as you'd like it to. Fine on servers, annoying on a 16GB laptop.
The Common Lisp problems you outlined don't really apply to Clojure, it's a modern lanaguage and has been stable and versatile for us.
Persistent data structures are really game-changing.
The ecosystem problem meant something different to me than most. Sure, when new tech came out (Kafka to pick a random example) there wouldn’t be an open source library ready for me. But I had stable infrastructure so I didn’t really care much about that.
The ecosystem problem for me meant that if something went wrong I was on my own. It’s a pretty unfortunate feeling to search for a bug and find nothing on Stack Overflow.
Par for the course in embedded. So isn't going to tell you why the kernel driver for the ABC123 chip is locking up on the XYZ15 eval board from Such-and-Such vendor.
You debug it yourself, whipping out JTAG debuggers and oscilloscopes, if necessary.
Because I work in embedded, I don't understand the whole ecosystem fuss.
I mean, joy of joys, you have the darned source to almost everything nowadays. That makes everything so easy, that all else is a footnote.
I can read source code (I do some security research for fun) and use a debugger, but I still feel like I'm programming by poking. What would you recommend to younger programmers to get out of this rut?
0: In this talk at the NYC Lisp meetup, Gerry Sussman was asked why MIT stopped teaching the legendary 6.001 course, which was based on Sussman and Abelson’s classic text The Structure and Interpretation of Computer Programs (SICP). Sussman’s answer was that: (1) he and Hal Abelson got tired of teaching it (having done it since the 1980s). So in 1997, they walked into the department head’s office and said: “We quit. Figure out what to do.” And more importantly, (2) that they felt that the SICP curriculum no longer prepared engineers for what engineering is like today. Sussman said that in the 80s and 90s, engineers built complex systems by combining simple and well-understood parts. The goal of SICP was to provide the abstraction language for reasoning about such systems.
Today, this is no longer the case. Sussman pointed out that engineers now routinely write code for complicated hardware that they don’t fully understand (and often can’t understand because of trade secrecy.) The same is true at the software level, since programming environments consist of gigantic libraries with enormous functionality. According to Sussman, his students spend most of their time reading manuals for these libraries to figure out how to stitch them together to get a job done. He said that programming today is “More like science. You grab this piece of library and you poke at it. You write programs that poke it and see what it does. And you say, ‘Can I tweak it to do the thing I want?'”. The “analysis-by-synthesis” view of SICP — where you build a larger system out of smaller, simple parts — became irrelevant. Nowadays, we do programming by poking.
For people w/ lisp dev XP, can I ask this: Is the ".emacs/config bankruptcy" issue a general problem? Specifically, when you use so many powerful devices, macros etc; but with little predictable code structure, then further development that sits well with existing code becomes pretty difficult.
A production codebase hopefully merits very different attitudes :-)
It's kinda fine if every library you use is _slightly_ different if you can have a unified interface for how to use them (spacemacs layers system for me).
It is annoying figuring out which invocation of variables sets the tab-width for _this_ new language though...
Also, most libraries in the emacs ecosystem have become more standardised in their interfaces, so things aren't too different nowadays.
But the global variable thing is interesting; Does regular lisp differ a lot wrt snippets having broad, global effects?
I don’t really care about libraries. I think generally it’s not very hard to write an ffi binding to a C library and most of the time you don’t need a library anyway. Most of the time you can just write your own library from scratch as you probably only need a few functions.
But maybe this is more a statement of the sort of programs I work on and if you worked on different systems you would want to use lots of external libraries that you don’t control.
I think the biggest issue with CL is how tightly attached it is to an outdated standard. It makes it harder to have things like good Unix support if your interface is through pathnames and streams.
Other thing which may be annoying about CL are the lack of a good way to consistently go from source code to a binary and that compiler optimisations can be unpredictable
Can you talk more about this?
We do use JSON but mainly for the product's Websocket API towards higher level SCADA system. As we have control over both ends there isn't much trouble really.
The project has soft realtime component also implemented with Lisp, and at least Lispworks exposes a substantial degree of control over GC. We however didn't need to tune anything thus far.
The system has perhaps a dozen direct dependencies to the libraries, luckily few of them posed any challenges in use.
There are many Common Lisp and Scheme environments. Many have excellent JIT compilers. They all have their relative strengths and weaknesses. So it really depends on your specific use case as to which environment has the best support. For me the biggest factor is that Lisp is very productive for use by a small and competent team. In those situations they tend to create a DSL and evolve the system. PG's essay describes the reason Lisp fails many projects in the section "The Blub Paradox".
There are so many things that can go wrong in a startup, it's not the time to pick a stack or database you know nothing about.
Most of my production code is python, but I also wanted to create a dsl so I used hy. You get to enjoy both worlds
I am a happy LispWorks customer but I also use SBCL (efficient code, easy to package standalone apps) and Clozure Common Lisp (fast compiler for development).
I recently switched to LispWorks after struggling a bit with CCL's Cocoa bridge and I wish I had done that sooner. A much younger, idealistic me would scream at the thought of it but sometimes you just need to throw some money at a problem. CAPI is really a killer feature.
Excerpt from the description: "These are the people who power the App Store, Apple TV, Apple Music, Apple Podcasts, and Apple Books. And they do it on a massive scale, meeting Apple’s high expectations with high performance to deliver a huge variety of entertainment in over 35 languages to more than 150 countries."
Lisp makes sense because it's a comfortable environment for interacting with my data. I'll be running the Lisp code "offline" so I'm not worried about runtime aspects like GC pauses and most of my I/O will be generating data for external tools (e.g. KiCad) in their own esoteric formats.
Coming back to Lisp it's nice to see that all my favourite software is still available and working fine, and there seem to be more active open source Lisp developers than ever. (Far fewer than Javascript, for example, but that's not my yardstick.)
> * poor ecosystem of libraries - few gems, most other half-baked
I actually had the choice for my libraries: DB interface, email sender, HTML template, data structure helpers… these ones with good documentation.
I have observed and explored the ecosystem for the last 3 years and I still find hidden gems. There are way more libraries than we can see at a quick glance.
We do a selection on https://github.com/CodyReichert/awesome-cl
> * poor community coordination
It looks like the oldest ones don't coordinate much and the youngers do (and push the established players, such as the CL foundation).
> * poor support for async
Maybe not for a full async runtime (cf Turtl feedback), but there is a good choice for all sort of async stuff (bordeaux-threads, lparallel, lfarm, channel, SBCL built-ins etc)
Clojure also has the core.async library. This can be used along with Reagent to build React-based apps; core.async allows you to do what Redux does but in fewer lines of code.
https://github.com/deckeraa/OpenStainer is an example of building apps this way.
Definitely no longer an issue for Common Lisp. Quicklisp is da bomb.
> Dependency management limitations with quicklisp
Never had any issues. It Just Works for me.
> have to restart the server every 20 days because of some memory leak [3]
You can get memory leaks in any language. You're much less likely to have a leak in a GCd language than a non-GCd language. But seriously, if you can't easily restart your server whenever it's needed, you have bigger problems.
> hack to tune GC [5]
I've run CL in production for many different applications. I've never had to tune a GC.
SBCL, SLIME, SLY, ASDF, quicklisp, and ultralisp
Other tools and solutions are available, but familiarity with this seemed to be what was required for comfortable use.
Ladder Insurance is a full stack Clojure shop and Watchful has a bunch of Clojure in prod too.
- There are lots of high quality libraries, but you also have access to the entire Java and/or Javascript ecosystem with very easy interop
- Not too sure what you mean by "community coordination", but I've found the clojure community to be exceptionally friendly, welcoming, and responsive. The core team is active and I've even had some questions answered directly by them in slack!
- There is some tooling built and maintained by the core team that makes managing dependencies painless
- There's a library (transit) that's widely used and well suited for marshalling data in and out of a clojure app. There's a library that's part of the core which implements the CSP (communicating sequential processes) concurrency model, similar to Go. It works in both Clojure and Clojurescript and is a very sensible way to manage async operations in a system.
- GC is managed by the JVM (for clojure), which is state of the art
[0] hylang.org
I doubt anybody knows how often I'm using LISP with this little tool because they only see the resulting HTML, but it's been a big help to me.
[1]: https://www.linkedin.com/pulse/building-cloud-choosing-lisp-...
Hmm this seems like something super specific to their async setup, rather than 'common lisp leaks memory'
Here i'll cover your points:
> poor ecosystem of libraries - few gems, most other half-baked
I'd say the ecosystem is high-quality. There are relatively few libraries (around 2000+) compared to other ecosystem, but on the other hand most of them are either interesting, or unique (wouldn't exist in languages that lack certain features), or really well-written. Or many of those qualities at the same time.
In the absolute sense, there are enough libraries, IMO.
If you need something that isn't available, you have at least three options:
1. Bind to a C library; for this, Common Lisp has many facilities/libs for this, some of them make this task rather easy.
2. Same as (1) but Use ECL (Embeddable Common Lisp) for integrating a C program with a Lisp program as a whole, easily calling C from Lisp and Lisp from C, etc.
3. Use the java ecosystem and use the Armed Bear Common Lisp (ABCL) implementation. This implementation makes calling java libs a piece of cake, basically you call a java method with just one line of code. Or instance a class with just one line of code, etc. Really easy.
> poor community coordination
That's a bit meaningless. There are really good quality books and resources, i think that's what counts.
> Dependency management limitations with quicklisp
Quite the opposite. Quicklisp is awesome, better than PIP or NPM. It "just works".
> And some specific red flags like:
> poor support for json[6]
The opposite, many libs for JSON. The one I use, I like a lot.
> poor support for async
What does this mean? You can use cl-async, but i wonder what would be the benefit when in CL you have actual threads and great libraries for concurrency like lparallel (which introduces parallel processing constructs) or chanL (which gives you an easy to use channel implementation).
Cl-async allows you to leverage the whole power of libuv (just like Node does), but why? I'd rather use threads.
> have to restart the server every 20 days because of some memory leak [3]
Never heard about such things.
Perhaps you are confusing the "condition - restarts system" (which is the Lisp exception handling system) with having to restart something?
> hack to tune GC [5]
The URL you cite says "We had to tune the generation sizes".
Any respecting developer who wants to release a high performance production quality service NEEDS to tune the GC. Tune the GC is not a "hack", is just configuration parameters. Ask any Java developer who has had to "tune" the JVM for best performance.
By the way, everybody raves about Golang but it has a GC system that is comparatively outdated to the ones in most Lisp implementations (or the JVM). For example CCL has a very good generational garbage collection.
Specific to common lisp: I don't find the ecosystem to be poor but I notice that many are turned off by the look and feel of library landing pages which do feel dated. In same cases I have felt the libraries lacked documentation, like Caveman2, but in the end I found there was little to document after reading some of the source code.
Specific to Clojure: I don't think this criticism applies to Clojure at all and the popular Clojure libraries usually come with great documentation as well.
IMO, JavaScript is also littered with mounds of crappy half baked libraries as well but people still use it for server side stuff even though they have a choice to use something else.
> * poor community coordination
Which lisp community?
> * Dependency management limitations with quicklisp
What limitations? Also, if you don't like quicklisp or common lisp in general, I think Clojure really nails dependency management and find it very easy to use.
> * poor support for json[6]
Specific to CL: I can understand this because there are a number of libraries and no obvious winner. JSON isn't the only data serialization format though. Guess it depends on what you need. https://www.cliki.net/serialization
> * poor support for async
Specific to Common Lisp: There is cl-libuv: https://quickref.common-lisp.net/cl-libuv.html and wookie.
Specific to Clojure: core.async is great and if you don't like it you are also free to use node.js as your runtime with clojurescript.
> * have to restart the server every 20 days because of some memory leak [3]
If I had a dollar for every node.js production app that suffered this very same problem I could probably retire today. I thought it was interesting that the author of that article says the service was re-written in node.js given node.js is hardly immune to these sorts of problems. IMO, async is just a pain to work with and these types of issues are a manifestation of that.
> * hack to tune GC [5]
Users in JVM land also report having to resort to tuning the GC or selecting the optimal GC for their use case. I don't think this is a reason to shy away from lisp.
> If you are using lisp in production for non-trivial cases, do these issues still exist?
I have not personally encountered these issues in Clojure or CL. That is not to say they do not exist. But they also exist in plenty of other popular platforms to varying degrees. See my responses above for examples.
> and, finally, if you had to re-do your project, would you chose lisp or something else?
I would still choose lisp, either Clojure or Common Lisp. I generally favor Clojure to CL.