1
We're now working on improving the scalability of LISTEN/NOTIFY in PostgreSQL, and to guide that work, I'd like to better understand how it's used (or was used) in real-world systems. What works well? What doesn't?
The current implementation has some known scalability bottlenecks:
1. Thundering Herd Problem: A NOTIFY wakes up all listening backends in the current database, even those not listening on the notified channel. This is inefficient when many listeners are each listening to their own channels (e.g. in job queues).
2. Commit Lock Contention: NOTIFY operations are serialized behind a heavyweight lock at transaction commit. This can become a bottleneck when many transactions send notifications in parallel.
If you've used LISTEN/NOTIFY in production, I'd love to hear:
- What is/was your use case?
- Does each client listen on its own channel, or do they share channels?
- How many listening backend processes?
- How many NOTIFYs in parallel?
- Are you sending payloads? If so, how large?
- What worked well for you? What didn't?
- Did you hit any scalability limits?
Feedback much appreciated, thanks!
/Joel
Questions for HN:
- What problems do you see with the approach?
- Any better approaches?*Dinesh*: (chuckling) "Look at Gavin go. He's not just burning bridges; he's torching entire countries now."
*Gilfoyle*: "I'm impressed, in a way. It takes a special kind of incompetence to unite an entire nation against you."
*Dinesh*: "Do you think he knows he's become Sweden's Public Enemy Number One?"
*Gilfoyle*: "Knowing Gavin? He probably thinks they're just passionate Hooli fans. The guy's ego could eclipse the sun."
*[TV shows a clip of Swedish workers striking]*
*Dinesh*: "I bet Gavin's going to try and fix this with another one of his 'brilliant' ideas. Like Hooli-branded winter coats for all the striking workers."
*Gilfoyle*: "Or 'Hooli Strikes Back' – a new VR experience where you get to fire union workers in virtual reality."
*Dinesh*: "I wonder if we can get the live feed of the moment he realizes he's screwed up. I'd pay good money for that."
*Gilfoyle*: "I've already set up a notification. There's no way I'm missing Gavin's face when a country full of polite Swedes tells him to go to hell."
*Dinesh*: "Poor Swedes. They were just minding their own business, inventing safe cars and meatballs, and then... Gavin happened."
*Gilfoyle*: "If Gavin can single-handedly shut down Sweden, imagine what he could do if he actually had a clue."
*[Both share a knowing look and burst out laughing]*