"Premature polyglot programming" is a disease that afflicts certain startups. While any sufficiently big company will be polyglot, a small startup needs to stay focused on a limited tech stack, for as long as possible. After all, each new technology requires someone on staff who has the skill for that technology, and when your startup is small, your talent pool will also be small, and so it becomes common that you only have 1 person on the team who knows how to run some particular technology (Kafka, RabbitMQ, Redshift, DynamoDB, MongoDB, Tornado, etc).
So you end up with a lot of single point of failures, where the "single point of failure" is the single engineer who knows how to run the technology that you've made critical to the company. Be wary. Avoid this if possible.
I notice, especially, as the tech industry developed better tools for managing complex devops situations (Docker, Kubernetes, Terraform) there was a tendency from some engineers to think "Nowadays it is easy to run 10 different technologies, therefore we should run 10 different technologies." Be wary of this.
Janet Carr's emphasis on simplicity is something we should all imitate.
Of course larger orgs will have higher tolerance for a wider set of technologies, but this tolerance is not infinite, and choosing tech because that one team was a fan creates staffing problems over time, especially as the original devs move on. Eventually you get “estate sprawl” that’s difficult to manage.
Choosing to minimise dependencies is definitely a (good) choice. The key is the right tech for the task at hand. Equally org-wide “use Oracle for everything” mandates aren’t helpful.
Balancing these at a large org definitely needs good engineering leadership!
I've been bitten by the ops hellscape of microservices and various tech many times throughout my career, and it's definitely shaped how I think about designing and building software now.
Some languages make it very painful to not have a mostly complete design upfront and then you’re trapped adding complexity just in case.
That said, I really don't think RabbitMQ is so bad. The default docker configuration for it is fine for most cases you're likely to come across for all but the biggest applications, and I do think that having the ability to have jobs restart/requeued when a worker crashes out of the box is worth the tradeoffs for slightly increased complexity. I usually just use a single docker compose and glue everything together in one big ol' YAML.
Still, there's obvious value in avoiding dependencies, so it of course depends on the size and scope of your project. If you think you're going to end up distributing this over 100 nodes, something like RabbitMQ or ActiveMQ might be worth playing with, but if you're just doing relatively small (at you appear to be in this project), it's probably the correct choice to mimic whatever behavior you need with core.async.
For ClojureScript, it was React. Immutable state / functional programming fits so nicely, and hiccup syntax (JSX as simple vectors) is the perfect declarative (yet transformable) DOM structure.
Wrt the original article, effect systems like missionary (which electric is based on) can be used to complement or replace queues, often resulting in simpler code.
I don't do a lot of frontend, but for anything that does involve frontend I kind of refuse to use anything but re-frame. It took a bit of dogma-acceptance, but once I did I really liked it for all the reasons you listed, and in particular that there's no special JSX crap, and instead just using vanilla vectors and symbols. This has the interesting advantage where you can generate your HTML logic without actually having to import the re-frame/re-agent logic, which can be useful if you want to decouple logic from rendering.
I haven't done any benchmarks, but I haven't noticed re-frame being slower than vanilla React, it seems to be a comparable speed, but as stated I don't do much frontend.
A real "whoa" moment comes when you realize you can also use core.async in ClojureScript :-)
I think my technical perspectives have moved in a similar direction: keep things extremely simple with minimum moving parts.
I maintained (automated upgrades) a RabbitMQ cluster and while it is powerful software it is operationally expensive. For a side project you probably just batch process in a cron.
If I were to take the approach in this blog post I would want everyone on the team to be extremely familiar with the model of task running: stuck jobs, timeouts, duplicate jobs, client disconnects and retries, stuck "poison" jobs seem like issues you might face.
I'm using a PaaS, so I didn't want to pay the extra money for a cron job. Maybe not a wise a choice, but here we are.
Totally agree about the issues I might face. I'm glad that the Clojure REPL is a thing, so I can test out all of a job's functionality before sending it off to async land.
It looks like Clojure has better RabbitMQ client options than Common Lisp, but still, very cool to build something on core.async and keep things cleaner and simpler.
