EDIT:
>Another of those tricks has to do with the fact that nearly half of our code is also written in LUA running directly on Redis.
Urgh. Ugly.
As far as putting a great deal of code into Lua scripts, we'll be putting together a few blog posts explaining how we make this manageable. We were able to double our API throughput by shifting this direction so for us this is worth the difficulty we faced in finding ways to modularize and reuse code in Redis Lua scripts.
My main beef with fsync every second is just that you will never, ever get a "this is what the server looked like when it went down". If the fsync at every query was only worse by a relatively small factor, and if you used write transactions for the majority of your writes (meaning fewer times needed to fsync for every query) which I'm guessing you are to protect integrity on writes, I don't see why this wouldn't be more appealing than fsync at every second?
A work around so far is to sync every 60 seconds on the master, and more frequently on the slaves. Another option might be to bump up the IOPS on the volume, but I believe that still isn't available on medium instances (which we are using as well).
I suspect we will play with this over time and honestly you have me curious now too what the throughput differential would be for individual persistence servers.
If it fits your ram of course (and no, swap space is not RAM, just don't)
But you can organize yourself, putting bigger data in the FS for example, and it should be ok.
The only issue with Redis is that it's much 'lower level' than other DBs so don't expect to do a 'SELECT * where condition' out of the box.
But someone may initially think that Redis can find all items that have 'cactus' in the text, which is not true.
One could say Redis is 'almost' a DB (as per the common conception of a db like MySQL, etc) and more like a 'build your own db' kit.
That would be about an EC2 m1.medium
It would be interesting to see how much it costs to run the whole cluster. I'd also love to know which part of the redis featureset they really use. Redis is great, but I think a lot of other database backends will give comparable performance when allowed to store all their data in RAM (MongoDB, Postgres, Riak, Cassandra, ...). The advantage of these (especially Riak/Cassandra) would be that for pure key/value semantics, they take care of all of the annoying operations overhead like rolling updates.
We are big fans of Lua (corrected typo) and much of the load is set, zset and bit operations. It's all the multikey operations (and the throughput we can push of those) that make Redis work for us.
Since we are pretty familiar/comfortable with automating infrastructure, dealing with clusters wasn't a big hurdle for us. This really wasn't a major deciding factor on our infrastructure design. Good tools and good Chef cookbooks make managing pools of servers relatively straightforward.
Could you describe these checks in more detail, please?
hset('shard.healthcheck', checkId, token) wait 500ms on every slave hget('shard.healthcheck', checkId)
Verify the tokens match.
Slaves are removed from the pool when the tokens don't, and allowed back in when the tokens match. Writes to master is disabled if a x/2 slaves are not available (where x is the number of slaves).
The drawback as I mentioned is that the patterns we found necessary are hard to retrofit onto an existing application... therein lies the rub.
Here is the redis author's blog post about that: http://antirez.com/news/52
My guess is that SSD's would be beneficial in ensuring the Append-Only-File and RDB snapshots are faster.
They claim to support Redis DBs of "unlimited size"... until they run out of ram in the cloud :)
