Redis re-implemented in Rust (opens in new tab)

You can contribute to:

https://github.com/frankmcsherry/timely-dataflow

https://github.com/frankmcsherry/differential-dataflow

Or, just tell your friends. :)

Better yet, write some python / pandas / dataframes / whatever_the_cool_kids_need layer on top, and rule the next big data drama cycle.

The more cores you have and the more RAM, the bigger advantage GC has. The thing with having lots of RAM is that it's very hard to take advantage of it with on-stack data (which can, at most, use about 1-2% of the total RAM available -- do the math) and thread-local heaps. Once you use thread-local heaps/arenas, you need to shard your data. Any cross-shard access would mean locking, which doesn't scale very well. That's exactly where GCs shine: they let you have scalable, concurrent access to data with arbitrary lifespan. That's why Java is used for those kind of applications -- it performs and scales much better than Rust can hope to do on large machines.

You are right, though, that if the processing is extremely "batchy" and all data dies at the same time, then it doesn't make a difference.

Little secret... with Rust you can just do big data processing on a single core => http://www.frankmcsherry.org/graph/scalability/cost/2015/01/...

Aside from project's reason, there's one very good reason to re-implement a bunch of stuff in Rust: testing whether it delivers. It's being pushed as a safer, better systems language. So, let's take many different C/C++ apps, rewrite them in Rust, and see what the results are across many metrics. By the end of it, we should know Rust's strengths and weaknesses in real-world coding.

The README answers this:

  To learn Rust.

Edit: It also mentions not being tied to UNIX and appears to claim it will run on Windows. That's certainly something.

>Could someone (or OP) elaborate on the value that re-implementing a whole software to a new language provide comparatively to just building an interface "bridging" both worlds?

Making it safer and even catching bugs in the original implementation (both things Rust will help with)?

Making it integrate seamlesly with the new language's ecosystem? E.g. Lucene is Java, and someone could use that, but there are tons of ports of it in C, C++, Python etc, providing convenience to integrate it with projects in these languages.

>And if I am not mistaken you would need a team at least as large as the parent project to implement new features, fix bugs and keep pace with it.

Not necessarily. A project with 10 part time contributors could be matched with a project with 2-3 full time competent hackers for example, or even surpassed.

This project specifically is probably not useful for others just because it is written in another language (unless it were to succeed in fixing problems or improving security or performance.) Redis is a server application, not a library, so there's a clearly defined and interoperable protocol to bridge the Redis server to other languages already. Writing a Redis client for the other language would be more useful.

More generally, as coldtea mentions, making integration into the rest of the language's ecosystem is the primary benefit of rewriting in another language.

The value of such a port to others depends on how easy it is to integrate between the two languages, either via libraries or other methods. The harder it is to integrate the two (and the absence of automated translation tools) increases the value of the rewrite to others.

Your Core-NLP example is actually an interesting one, because that library has already been ported to other languages... It is available for the C#/F# ecosystem (http://sergey-tihon.github.io/Stanford.NLP.NET/).

In this case, it's a learning exercise. Sometimes, it's because there are other benefits, such as making it easier to distribute or easier to tie into specific parts of the algorithm than may be possible by calling a library that provides a high level interface.

From a learning perspective, reimplementation has key advantage versus other kinds of projects: the design is completely done, so you can focus exclusively on the mechanics of implementation.

By the way, if you are looking for a production-quality Windows port of Redis, there is a fork available at https://github.com/MSOpenTech/redis. We (Microsoft) provide it in production as Azure's cache service today, and are committed to continuing to work on it.

Also notice that the main developer only has two months of experience with Rust, so it is probably not as stable and well tested as Redis. As stated in the README, the main goal is to learn Rust.

The README does seem a bit ambitious, but it notes that the purpose was to learn Rust.

I'm sure pull requests to bring it up to feature parity would be welcome!

The phrase i like most is "data structure server". It's basically a giant heap that you can fill with data structures - strings, lists, sets, sorted sets, maps, bitsets, and this wacky HyperLogLog thing:

http://redis.io/topics/data-types-intro

The data structures are all addressed by string keys.

Redis can persist this heap to disk, and load it again, so you get a measure of durability, but the typical use case is for data you can afford to lose - caches, metrics, some kinds of events, etc.

Redis's key non-functional strengths are speed and robustness. Operations people love it because you stick it in production and it just quietly keeps on working without needing attention or setting your CPU on fire.

To my mind, any project should have PostgreSQL as its first data store. But it should probably have Redis as its second, when it finally has some data that needs to be changed or accessed so fast that PostgreSQL can't keep up.

(Kafka is third)

From their official github page, Redis is an in-memory database that persists on disk. The data model is key-value, but many different kind of values are supported: Strings, Lists, Sets, Sorted Sets, Hashes, HyperLogLogs, Bitmaps.

It simply means that the key-value store is directly loaded into the memory (RAM) and is available for fast access, but the data is retained (persistent) even after the application is closed.

It is usually used as cache store, queuing messages to communicate with different processes locally or distributed.

It's a memory cache to take some load off your DB. The good thing about Redis is the useful data structures it supports: lists, sets, hashes, bitmaps and somewhat more specific sorted sets.

Put key & read key with straight-forward abstractions. Simple and beautiful.

It can be used for caching, queues and for applications with volatile data.

> but I'm wondering if at some point a Go version would perform better than C.

The answer is "no" with a certain amount of probability. Redis isn't single threaded by lack of capability, but by design. Concurrency for multiple CPUs will actually slow down a lot of the stuff you see, as you will need to introduce locking mechanisms.

Also, garbage collection is highly tuned and customized in Redis to the use case of an in-memory-DB (in stark contrast to usual allocation patterns of an application), up to the point where it's almost impossible to replicate the performance in a garbage collected language.

I love Go and we're a 100% Go (and Angular) shop, but for an in-memory DB it wouldn't be a sane choice.

You can turn off disk persistence in Redis. My main usage of Redis is with disk persistence turned off (it handles a few hours worth of samples of data that we don't care if we lose - we care about having long term averages only of the data in question).

There should be minimal overhead from having the capability in Redis due to the way it implements disk snapshots (RDB snapshots are done by fork()'ing and relying on copy on write to let the main process keep on doing its thing while the child process writes the snapshot, so the main process doesn't need to care; other than that Redis offers logging/journalling of changes, but the cost of having that as an option is trivial if it's switched off).

Having pluggable backends for things like Postgres or DynamoDB seems a bit at odds with the purpose of Redis, which is exactly that you pay the (very low) cost of in-memory manipulation of simple data structures, though if a single Redis server could partition the key space between plugins, it might potentially be useful by letting you e.g. move keys between backends and still access them transparently to the client. E.g. for the samples I mentioned above, we roll up and transfer data to a CouchDB instance for archival now (doesn't matter that it's CouchDB really - we just need a persistent key-value store; Postgres or DynamoDB would also both have worked fine), but if I could archive them while still having them visible in a Redis-like server together with the in-memory keys, that'd make the client a tiny bit simpler.

For most Redis usage, I think paying the cost of connection setup and teardown and sending HTTP headers etc. would likely slow things down immensely. At least it would for my usage. Having a HTTP interface as an addition might be useful in some scenarios to enable new use cases, but as a replacement for the current Redis API would be a disaster.

If you want to explore alternative interfaces, I'd instead suggest going in the opposite direction, and experimenting with a UDP interface. In a typical data centre setting packet loss is low enough that while you'd need retry logic, it wouldn't necessarily get exercised much in normal situations.

(On the other hand, for the typical request/reply cycle it might very well not give any benefits vs. tcp in most scenarios where multiple request/replies are done over a single connection and thus amortising the connection setup cost - would be interesting to benchmark, though)

Ah, HTTP is your hammer

What I'm personally really surprised about is that nobody's rewriting Redis as a unikernel to clear away all the OS context-switching/networking overhead from its basic operations.

The "Why" is @seppo010's to answer (but having it run as is on all OSs is a big plus for one). As for writing it in Assembler, that makes little practical sense since Redis is written in (ANSI) C and it quite well optimized. In fact, if you profile Redis you'll see that very little time is actually spent by the code itself - OS, storage and network are the real bottlenecks usually.

Its at the very top of the readme. Here is a direct link to your question. https://github.com/seppo0010/rsedis#why

That's a fair criticism, and well taken, but keep in mind I'm learning the language as I go, and most of the commits are just rewriting things because they were suboptimal, not idiomatic, or hard to read. At this stage, I would consider adding comments wasteful.

I also have no intention of making this project live as long or have as many users as Redis does.

Yep, works with 1.0.

He is at least curious. https://twitter.com/antirez/status/611189939519229952

He links a list of missing stuff in the readme.

And if you read "Why? To learn rust" and ask "should I use this in production"...