I know that part of the design goal of Redis is to create a system that is "Simple" and "Readable", and the redis server as it stands succeeds at this goal admirably. But the approach in this draft is neither particularly simple nor is it going to make the codebase more readable. It seems fairly awkward and introduces more of a burden on the operations and management of your product.
Can someone explain the value of this design?
The idea is to provide a system that is built into Redis and does not require external dependencies, and that exploits Redis-specific things to do the work better or in a simpler way.
Sentinel (already partially implemented, and you'll see a beta in mid July) uses Pub/Sub to discover other sentinels, INFO to discover attached Slaves, and other Redis-specific capabilities as well.
It is created as a special execution mode of Redis itself, and has a memory usage of 1MB, with a CPU usage that is near to zero. This means that running a Sentinel is basically for free, you can put one everywhere.
It is written in C as a "sentinel.c" module of Redis. In fact you can think at Redis itself as a framework to write event-driven services. So there was a lot of code reuse.
At the same time being every Sentinel a Redis instance, it understands the Redis protocol (but a different set of commands) and you can query every Sentinel for the full state. Or subscribe to a channel to get events that are happening using Redis Pub/Sub primitives (events like: this master is down, or it was failed over, use that instance instead).
This are a few of the reasons I designed and I'm building a new system.
Currently I'm at 50% of the work after few weeks of work, but actually a lot was done in the latest 10 days (before there was still too much 2.6 work). I think with another few weeks we'll have a working product.
What this doesn't answer is my first question: why do there need to external special-mode sentinel processes? Why not have the slaves negotiate themselves? You're already doing a bit of this work, wouldn't it be simpler?
You say the sentinels are "for free" and they may be extremely inexpensive to run; but they're not "for free" from an operational context. They're another process to provision, start, monitor, and consider.
One thing that worries me... normally if I have a configuration setting that might change at runtime I store it in redis! Does anyone have a good way of storing the configuration of where the redis server is in a way that can be updated at runtime (assuming a standard shared-nothing architecture) - without putting it in redis?
Modify clients configurations when a slave is elected.
(I assume elected == promoted). This is an idea I haven't really seen in other servers/services before, and I'm curious how this will be implemented. I assume some sort of pub/sub subscription to each of the slaves so that your server is notified when one of them takes over? It sounds tricky, but really interesting. Document seems a bit scant on details for this part at the moment.
Regardless, really thrilled about this project.
[Edit: Ah, I missed this part:
client reconfiguration are performed running user-provided executables (for instance a shell script or a Python program) in a user setup specific way.]
