Don't add a message queue until you really need one. And if you do make sure you account for it being down, running locally, back pressure if the queue gets full, monitoring, and logging at least.
When I try to estimate ROI on those features it's usually miniscule. A/B test are unconvincing and the aggregate reports synthesized from the data look more like numerology than rigorous science. More-over, senior leadership usually doesn't realize how much additional liability they take on by collecting and storing so much customer data.
It's not 2010. The large tech firms are your direct competitors and regulators across the world have caught up, including in many US states. If you have a primary product for which people will pay money, then the "surveillance agency as a side-business" model of the 2000s-2010s is probably a bad idea. It's a net expense that exposes you to liability and distracts your team.
Additional benefit: without real-time analytics and personalization, you can go even further without a message queue.
That's not the architectural reason for message queues in my experience.
Primarily, a message queue is used when there is a potential for a bottleneck in the overall application throughput where there is high cost of reacting to some event so instead of reacting to the event synchronously, you queue the event and react to it asynchronously.
Some very common use cases:
1. Your UI. Operating systems use a message queue to capture and forward inputs. Your mouse and keyboard events are all queued in a message queue.
2. Webhooks. Many webhook origins have timeouts on responses and your application must respond within a given window or it will be throttled or downgraded. In this case, the best practice is to queue the event to be processed asynchronously (that queue can be a simple database table with your own logic and wrapper around it or something like AWS SQS, Azure Service Bus, or Google Pub/Sub).
3. Mitigating Throughput Bottlenecks. Most applications have a read/write asymmetry so it makes sense to optimize your architecture to scale for reads. But what if your application occasionally has to handle a burst of writes? Should you size your infrastructure for that case? One approach is to proxy the writes through a queue so you can size the infrastructure for a maximum throughput that is managed by the queue. For example, instead of 1000 concurrent writes per second, a queue can capture the write mutations and trickle out only 100 concurrent writes per second. Instead of sizing your application to scale to handle 1000 writes per second, you only need to size your queue to handle that scale.
4. Resiliency. If a message fails, it can be retried according to whatever heuristics make sense for the domain. Sure, you can use a simply loop to retry, but every message queue provides some mechanism for handling retries, failed message delivery, and so on. If you decide to roll your own and log a failed call into a database to try it again later...well, you've effectively captured a message in a custom queue.
So, you can go very far without a message queue... by using a simpler message queue?
All of this to avoid spinning up an SQS queue?
i’m not sure if this pattern can be called a type of saga.
you basically have a tiny queue in you db using a special table that you drain to an external queue server.
What you might mean is that you might not need a complicated server setup e.g. Kafka for simple message queues?
A real queue have an API similar to JMS, ex rabbitMQ, IBM MQ, Microsoft MQ, …
The difference is you have a mailbox of message from which you remove messages you have consumed by acknowledging reception. Message will be sent to subscribers until they are acknowledged.
So publisher write message m1, m2, m3
subscriber receive m1, m2,m3 and ack only m3.
it will later receive m1 and m2 again but not m3.
Kafka is more like tcp It’s a stream.
Then most of your statuses are Complete and finding an ever decreasing percentage of Incomplete rows will result in essentially a table-scan.
Then that table scan will start to take longer than your cron interval or will biff through your IO or CPU and you've got problems.
Then you might try adding indexes or beefing up your queries and then you'll realize you've just traded 6 for 12/2 because now your index updates on writes are hurting you more than your scans on reads.
Then you'll have the brilliant idea to cull your data regularly or have different tables for Complete vs Incomplete.
And then you'll want to do this culling or row-move in a transaction because you've never bothered making the job idempotent because it was implicitly so in the database all along, and now your transaction locks will compete with your cron scanning or index updates and now you'll maybe put the status table(s) in a different database and work on idempotency.
And then maybe you'll take a step back, have a nap, and wake up to realize you've just implemented a queue but not very well and have tied its design to your specific business-flow and good luck adding retry logic or new status fields.
If you're gonna use a database as a queue, make it a different database server (or schema!) from your business data. Don't tie business transactions (completing the job) to database transactions (since ultimately there will almost always be more than just one data/processing system involved), and maybe really just adding a simple Redis or SQS thing from the beginning isn't going to cost you that much so might as well squeak it in while the costs are still low.
It's like you showed up just to dump on the author. You added no value here.
You fail to articulate why it was hard for you to read. You failed to articulate what might have been better. You failed to have a basic level of compassion and human decency in a way where you could even make a base-level attempt at wording your post as anything other than a snarky attack.
I'm personally sick and exhausted of commenters who behave like this. The people who contribute nothing but put in the extra effort to show up in comment sections to dump on people who are trying their best to share or build something.
I didn't know I was trying to sell a message queue, thanks for letting me know!
I don't get it but there's a good chunk of people who are ready to jump on anything that looks even slightly comercial by even one single aspect. I've been told my personal blog was AI generated blog spam once in the past when I had literally nothing to sell. I also don't understand the fundamental issue with having a commercial involvement as long as you aren't trying to BS people or shove advertising down their throats.
My personal opinion is that if you don't want to come off as selling something because your genuinely interested in tech, just publish under a different domain or pseudoname
> This works reasonably well. The polling period of reading the events may become a problem, but you can make it better by using database triggers.
to
> But when we look at tailored backends in the open, especially in architectures for FAANG, MAMAA, or whatever the acronym is until the next time Zuck decides to pivot Facebook, things look different.
unsatisfying.
Somewhere I'd love the walkthrough of in what scenarios exactly does the database-as-event-store+ad-hoc-worker architecture fall over rather than immediately jumping to what "the pros" do. That's more of an appeal to authority rather than something I can learn from.
Not saying you need to do this in your article but if not at least a link to read more would be welcome. :)
I thought of that myself to be honest but the article was becoming too long and decided to cap. Great point about the link, that's great feedback!
I've often thought there is an analogy with broken "real-world" processes where the same thing happens.
And generally this system should not use the message queue to just get things resent - because most likely in this situation you still need to be mostly handling new messages rather then old ones when you're stuck trying to catch up.
All of which tends to be a product of just utterly magical thinking from managers and solutions architects as to how anything works, and how likely an outage is (basically guaranteed no matter what anyone says about their product).
I can't find the reference, but consistent with that, I remember reading someplace that the number of items in your queue should typically be zero, since queues can't actually make the rest of the system any faster. All they can really do is buffer temporary bursts, or buffer while the rest of the system recovers. And for that you'll definitely need your recovery system working as a prerequisite.
Depending on your requirements you either stop the state of the whole system or use something like dead-lettering and/or delay-queues + retry count headers to make sure that you can reprocess failed messages without letting them accumulate until you're wasting CPU processing messages that will never work.
The other problems with message queues is that you need to make sure you have alerting on all your queues otherwise you're just waiting for trouble.
There's sender channels, receiver channels, xmit queues, local queues, remote queues, dead letter queues, topics, subscriptions and all kinds of other things to learn about. Add that to the crappy IBM tools and their documentation that is full of dead links and you're in for a lot of learning through trial-and-error.
2nd'd & 3rd'd ...
You hired a 50 man lumberjack crew to cut down a sapling.
I've been spending a some time to remove IBM MQ out and going to "simpler" queuing solutions. First of all, cost. Guess what, we're not getting anything out of licensing per year to justify the cost.
With improving the end points, we don't need all that complexity. With 5 9's uptime on the network and smarter end points, retrying messages isn't expensive.
As always, YMMV.
Would never ever architect an app in production without leaning heavily on queues because it makes so many of the hard parts trivial.
To put it in another way: if I were to build an (fairly simple, monolithic) API, would I automatically benefit from using a queue? If yes, how exactly would a queue be used, and in what way would it bring a benefit?
Queues basically serve as a decoupling layer. If you need one, you'd benefit from one.
Example one: you put incoming requests into a queue. If latency isn't a big concern, that means that your service can go down / be restarted for a moment while the queue buffers all the incoming requests.
Example two: you write a HTTP API for an interactive web application. Fast response times are important, so you do only what you absolutely must inside the request handler, and everything else that can be done later (like sending mails) is put into a queue, and can be dealt with later either by another component, or by your service running "queue worker" mode instead of in "http responder" mode.
If your services is just reading data from a DB and handing it back to the user, you usually don't benefit from a queue. But most apps tend to become more complicated with time...
They aren't _queues_ per say but they have a lot of the same DX. Depending on your runtime environment as well, you might benefit from splitting side-effect jobs from your API request/response cycle. That way you can still respond quickly while enqueueing work (locally) all still within a monolith.
Disclaimer: not affiliated with ZeroMQ, just a heavy user of BSON and ProtoBuf that goes with ZeroMQ.
In fact, ZeroMQ library (libzmq) supports over 13 different methods of queuing.
https://stackoverflow.com/questions/50656232/guaranteed-deli...
echo '{ "symbol": "AAPL", "bidPrice": 174.1 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "bidVolume": 1000 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "askPrice": 174.2 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "askVolume": 2000 }' | rpk topic produce book-updates echo '{ "symbol": "AAPL", "bidPrice": 174.0 }' | rpk topic produce book-updates
However some top of the book update can result in an entire level disappearing. In that case, both price and volume changed actually at the same time, yet according to this example it will be broken down into two messages? This will lead to inconsistent state of the book when I read it between two messages. Or did the example just happen to show one field per message, and multiple fields per message are possible too?
Easy to install, sane config out of the box, developer friendly definition, relatively flat learning curve with highly googleable answers, and widespread support for AMQP.
There may be something faster/leaner or with some bells and whistles, but if you just need message queuing it's hard to beat.
Getting RabbitMQ to work in a cluster is a nightmare.
Be aware of that when selecting it as your message queue :-)
If your exchanges, queues, routing keys, etc. are planned out ahead of time it should not be difficult to manage.
Sorry what? How do they make so much sense? The article gives a full backstory and diagrams for event sourcing pattern it says every one knows but has no description as to why a message queue makes more sense than event sourcing?
I am so fed up of reading half way through an article just for the author to throw away their attempt at being informative and go straight into a sales pitch.
I think I am just going to say event sourcing makes so much sense and call it a day, I clearly don't need their product.
The jump from "saving events to be processed works pretty well" to "you should use message queue" wasn't explained at all.
I use redis to keep a store of last updated timestamp property (that is, the last time the data was updated), and I essentially throw out a message if the timestamp is older.
For any writing changes I make I have a validation loop in the code path as a secondary validation.
I think once you get used to been defense about incoming messages you can handle most delays
(We work with a design firm that provides illustrations for the blog)
