An unexpected Redis sandbox escape affecting Debian-based distros (opens in new tab)

I didn't go there at any point while writing the article, because, for a Linux distro (i.e. something that tries to be all things to all people), I think Debian does a great job, and the packaging policy is a net positive, by a wide margin. These things happen sometimes, and Chris Lamb (the package maintainer and Debian / reproducible builds extraordinaire), as well as both the Debian and Ubuntu teams, handled them very well.

When running large scale production services, the best practice is building your own image, perhaps starting minimal Debian or Alpine core, using automation, and the tech giants have settled on statically linked binaries a while ago, so Debian packaging "idiosyncrasies" should not apply anyways.

Distributions continuously patch packages. It's necessary to be able to ship a working, integrated distribution at all. It's also necessary because users expect some level of consistency and integration between components, and distributions are the spaces where that can be achieved. Usually these achievements then get upstreamed so it's easy to forget about their origin. There's also the cases where distribution users just fundamentally disagree with upstream policy and use their ability to change Free Software to make it happen - such as the disabling of auto-updates in distribution packages coming outside the distribution packaging system, or the disabling of [opt-out] telemetry.

So there are many good reasons distributions patch. It's not reasonable to paint them all with the same brush. If you want to credibly criticize this, you need to be more specific.

Because all of the issues they've genuinely fixed, security and otherwise, are meaningless.

Or are you just basing this on <bad thing> happened, thus all other cases (hundred of thousands of patches) are wrong?

The real bug here is not having a test for this, or not running the tests as part of the packaging process. We should not have situations where packages are so sensitive that a small change to them should be considered unsafe. Open Source is pointless if your source is regarded as immutable. If there is a subtle detail like this (and sometimes it's unavoidable), it should be covered by a test so that ensuring the end result is "ok" is nothing more than seeing the tests pass.

I'm not going to weigh in on which party should be responsible for that test.

If you think the patches are random, you must be missing something.

If upstream developers took good care of their security Linux distributions would not have to do all the patching and dependency unbundling.

Maybe upstream shouldn't make their security measures so fragile that everyone downstream has to resign themselves to not touching anything. Open source software is supposed to be modified and adapted.

Is this Lua sandboxing really a good idea? It it robust? It didn't work out well for Java, which removed the SecurityManager feature recently. Python gave up on attempts at sandboxing many decades ago.

Yes, of course it's vulnerable, verified with Docker debian:sid. That was my first reaction when I read this, but I wanted to verify it first. You beat me with this post.

Since you've already let the cat out of the hat (which is not ideal), please file the bugs at Debian and Ubuntu.

Test command:

    redis-cli eval 'return select(2, loadstring("\027")):match("binary") and "VULNERABLE" or "OK"' 0

While we're at it, redis has ignored the advice at: http://lua-users.org/wiki/SandBoxes Almost all of the critical functions (loadstring, load, getmetatable, getfenv, ...) are present and unprotected in the redis 'SandBox' (which isn't).

Which means, disable scripting or shut down your redis instances NOW, which do not run with the same privileges as any client which has access to this. Scripting can be disabled by renaming the EVAL and EVALSHA commands to unguessable names.

Given that lua is meant to be an embedded language, its kind of surprising this is neccesary.

Don't you know? Dynamic linking is always better for security. This is a great example. With dynamic linking you can more quickly patch security holes caused by dynamic linking.

Dynamic linking is Debian's preferred approach. When it works as intended (which is usually), it's generally a good thing.

If everything were statically linked, getting your daily security update would basically involve redownloading the entire distribution, since when a core component were patched everything would have to be rebuilt. This wouldn't be practical. Therefore, dynamic linking is the norm on binary distributions.

Debian really hates code duplication and prefers risking security issues resulting from trying to reduce code duplication. This is not the first time this sort of thing has happened, and has caused security issues.

Unnecessary static linking is a Policy violation.

That's what I'm wondering, too, right now.

It's trivial to DoS-hang redis with the script feature (and SCRIPT KILL won't help).

And I found at least 3 DoS-crash, because it hasn't backported fixes to its copy of Lua 5.1.5 (but Debian's liblua 5.1 might -- I haven't checked).

And that's without even exploring the really problematic builtins it still has available.

Maybe they should instead clarify their security guarantee for redis scripting (e.g. "none").

I think your edits are mangling the intended meaning of the quote. Your script should not try to perform other system calls, because it won't work.

If I understand correctly this RCE is mostly a big concern to hosts that don't make use of redis authentication, as eval is locked behind it? Some quick testing on my end suggests the eval command is parsed before authentication check but requires authentication for execution, but I may be wrong here.

In a perfect world we can all update immediately but when we still have some servers in Ubuntu 16.04..

oof, and just for people who are curious but this affects Ubuntu / Ubuntu server as well being a Debian derivative.

Right -- you can escape the sandbox that Lua runs in, inside Redis, and get the Redis process' user to execute arbitrary code in its environment. Redis might be inside a hardened container, or running as a privileged account on a shared machine.

redis-server is just a plain executable. Unless you're putting it in a sandbox yourself - for example, using a VM or systemd features - redis has no sandbox. There is nothing to escape from.

The Lua interpreter _within_ redis acts as a sandbox, which got a bit mangled here. Most features are not implemented in Lua, though. So this is only used for things like the EVAL command.

Upstream Redis doesn't. At least in my builds from upstream, luaopen_package and luaopen_os aren't even available on the binary. Debian does things differently, though. They need package in order to require the "json" and "bit" libraries, which are packaged separately on Debian.

Another thing that would be interesting and affect upstream as well would be getting the "debug" package back from the redis error function.

The only case I had problems with unattended upgrade was with redis... Both the master and the replica restarted at the same time, which caused some issues.