Even though I am a big fan of go, I've personally built two container runtimes in other languages do to the namespace clumsiness.
Personally, I think rust is an excellent alternative for namespace utilities.
EDIT: there is more information and links in the issue in the netns library: https://github.com/vishvananda/netns/issues/17
Disclaimer: I am one of the original libnetwork authors and we have been aware of this issue with go for some time now.
Can you share any details on those other container runtimes?
At the moment I'm incredibly busy, but later this year I might be able to start working on that too.
The Linux syscall interface exposes certain functionalities that are much more easy to reason about at the process level such as namespaces, capabilities, seteuid and so on. However these syscalls all operate on the thread level (since the kernel treats threads pretty similarly to processes). Therefore in order to perform these operations safely you need some sort of process wide mechanism to apply the operation on every thread (and don't forget error handling!)
This is _not_ just a golang problem or an M:N threading problem as many comments suggest. The kernel really needs to provide new syscalls for these features that operate at the process / thread-group level. The current syscalls are extremely difficult to use correctly in any multithreaded context in any language. When you consider the security implications of these features it makes the problem even worse.
Check out https://ewontfix.com/17/ for a really good analysis of the difficulty musl libc has faced making a multi-thread safe seteuid on Linux. There are also many bugs in glibc related to this as well. Linux makes userspace responsible for patching up the leaks in the kernel's process abstraction and that's really not a job that userspace is in the right position to take on.
Or it could provide another clone flag that indicates that threads spawned that way should share privileges and similar things, then runtimes that need threads to all behave the same way can opt into that. I suspect that some tools do advanced privilege kung-fu that relies on those per-thread properties.
This is hardly a linux specific issue. Prominently for instance pthread_setugid_np exists on OS X, threads for different subsystems exist on Windows etc.
I've hit this exact problem with multithreading in C and setuid and just because it _can_ be managed in C doesn't make it easy or straightforward.
Therefore, I mirror the sentiment that there needs to be a way to operate on a process level, even if that has some interesting consequences.
(P.S.: In C, if you're using glibc, it DOES actually patch this issue up on its own using one hell of a nasty hack.)
I too love reading the code for nptl(7). It's a riot. :D
I mean, what's the better alternative to Go for this work? Maybe Rust? It is, at least more controllable at a lower level...but, not as easy to pick up for people coming from a C/Python/Perl/Ruby systems and ops background. I'm not saying one should use Go for containers/namespaces programming, but a lot of people are with some success (probably also banging into the namespaces issue now and then), I'm just saying it's not obvious to me what the better alternative would be.
This issue is one of the downsides of Go's M:N scheduling. The OS is simply not aware of what the Go runtime is doing, and as a result you get impedance mismatches like this.
It "raises a few eyebrows" because M:N scheduling is unpopular outside of Go and Erlang. It was tried early on in the Linux world and abandoned precisely because of issues like this. Go has repopularized M:N lately, and it's proof that such a system can work for lots of apps, but the downsides of that decision are every bit as real today as they were in the early days of NPTL.
But, are you saying that to switch to a namespace in Go one must fork, whereas one wouldn't need to fork in C unless you need to switch namespace and you need concurrency (because you can determine when things will happen with precision in C)? I don't know. Again, this is beyond my understanding of Go right now.
I may just not be understanding the implications of this. The code to deal with it looks reasonable enough to me; it's a smallish function, easily isolated. I think the author did a great job explaining the problem, the troubleshooting, and the solution. I just didn't see the problem as being all that damning of Go...but, that may be a reflection of my shallow understanding of the problem, or of the implications of the cost spawning a new process (to me, I always think back to the old adage "fork is cheap on Linux").
Arguably it has been implemented with clone(2) which has flags.
> in fact, it's easier to reason about fork than something like POSIX threads, IMHO.
Not if you call fork() in a multi-threaded program. That ends _exceptionally_ badly (let's just say there's a reason Go doesn't expose syscall.Fork and it has to do with horrific deadlocks).
> I just didn't see the problem as being all that damning of Go.
The problem is more subtle, and it comes down to maintainability and understandably. If you ever decide to read the runc codebase, I apologise. One of the reasons the codebase is so scattered is because of these sorts of hacks where you have to work around issues in the Go runtime (because it doesn't give you enough control). In the article, whenever you read the small function you have to keep in mind that it's actually spawning a subprocess (which then means you have to think about what namespaces had the process joined and so on). Go is an okay language, but it simply wasn't designed for stuff this low-level. We would be much better served with Rust in my opinion.
> But, are you saying that to switch to a namespace in Go one must fork, whereas one wouldn't need to fork in C unless you need to switch namespace and you need concurrency
In C you don't need to fork to switch namespaces, you just call setns(...). For the PID namespace you need to fork, but that's just a quirk of the interface.
In Go, you theoretically don't need to fork either (syscall.Setns is available). However, there is no real way to safely use it. First of all, the namespace interfaces in Linux are quite fragile when it comes to multi-threaded processes, but combine that with a runtime that will switch you between OS threads at random. And while the documentation on runtime.GOMAXPROC and runtime.LockOSThread might trick you into believing it's possible to stop the Go runtime from doing a clone(CLONE_THREAD|CLONE_PARENT), you can't.
The article is stating that to control the namespace that the threads execute in, that a separate process must be spawned so that the entire process can be forced into the correct namespace. Otherwise, the runtime can spawn new threads as it sees fit and you don't have control over which namespace they are in.
It might be possible to work around this from within the same process if it were possible to force the runtime to not spawn new threads in particular cases. If you could control when the runtime was allowed to spawn new threads, then you could organize the program / threads in a way that would keep the correct code operating in the correct namespace. Unfortunately, you can't.
Disclaimer: I don't know Go, but this is my understanding from reading the article.
When changing namespace happens often in a hot path, forking might not be fast enough.
I disagree with Walton that the blame is on the N:M threading. It's really on the Linux kernel that has never made a clean distinction between threads and processes, both in kernel and in the APIs.
In addition fork() style concurrency was essentially nonexistent outside of Unix.
This is not a problem of M:N. This is a problem of Go being badly designed. Not new.
Due to the implicit M:N mapping, Go goroutines are extremely cheap. This allows you to spawn as many, as your algorithm naturally requires. The Go runtime will automatically map them to native threads - typically one per avaliable CPU. As a consequence, a Go program that heavily uses goroutines has a pretty clean code and scales without much overhead across a large variation of number of CPU cores.
Although they have their own set of issues, Windows fibers are also barely used.
Let me tell you how runc works. runc is written in Go, and we take an OCI configuration file. Because we can't just fork and set up all of the namespaces in Go, we have a C function called nsexec which is specified as __attribute__((constructor)). This ensures that our code will execute before the Go runtime boots. The parent process writes (using netlink as the wire protocol) to a pipe that the child has open and is parsed in C. Then, the child will have to do a series of forks, unshare, setns, {open,read,write} and so on (and the final PID needs to be sent back to the original parent) in order to set up and join all of the necessary namespaces.
In C, this code would be _immensely_ easier to read, write and maintain. Just look at LXC. Personally I really wish people had just gone with Rust earlier on rather than implementing everything in Go. I've had nothing but pain from Go.
Is it merely fear of C that keeps so much of the container infrastructure on Go? I've only spent a couple of weeks looking peripherally at Go, and I already like it better than C (which I've poked at peripherally for ~25 years), but I don't know it well enough to know its warts.
To avoid the authors' issue, you can write some functions in C. CGo has a very high level of integration (you can mix languages in the same source file) and would be quite simple for the case of a setns/execve wrapper.
No, no it doesn't. CGo is slow as balls to call into and return from.
Types can't fully be shared.
It's interacted with via comments.
You can't use cgo to control the threading of the Go runtime itself, which is the real problem here.
The behavior you want is to be able to call linux syscalls that operate on threads and not be utterly fucked. That behavior cannot be accomplished with go nor go+cgo easily.
Threads in cgo are also kinda fucked.
Go is deliberately opinionated about threading of the runtime. I think it's unlikely Go will offer much more control over these internals (beyond GOMAXPROCS), given the philosophy around e.g. GC tuning.
Cgo is a beefed-up runtime.LockOSThread() that could be used to avoid having the author resort to a helper process.
> CGo is slow as balls to call into and return from.
They're slower than Go function calls, but they still take only nanoseconds. This is negligible on the authors' scale of "launching an entire container".
For their case of "it is not possible to guarantee that a new OS process ... will run in a given namespace", you're not even returning from Cgo after exec.
> It's interacted with via comments.
Do you use build tags? Or go:generate? Like it or not, it's idiomatic.
I'm with you on the types, though! `go tool cgo -godefs` helps, but it would be great to see improvements especially in the reverse case of exporting Go buildmode=c-shared for C consumption. Still, a little marshaling seems tidier than a whole helper process.
Separate processes, like the post suggests?
I have no idea why someone would expect user-level pseudothreads to execute across system-level primitive boundaries.. seems fairly obvious to me.
I don't expect a chrooted daemon (e.g. apache, etc) to have access to parent thread contexts.. etc.
Fork & pipe IPC shouldn't be too difficult for anyone to understand, beyond that, if you don't understand these things, you probably shouldn't be writing code that complex..
runtime.LockOSThread() does exactly that [1].
Trust me, we've been trying to get Go to co-operate with containers for quite a few years. It's not as simple as just reading the standard library docs. ;)
I think C++ may just be too rich a language to be a part-time thing. I think Rust might have the same problem (though I'm finding it easier to read than C++, it doesn't seem to be afraid to require a lot of learning time from its developers). But, I'm willing to entertain other theories.
For whatever reason, Go seems to have very quickly entered that category of language that systems and ops people are comfortable with.
C++ looks good on paper. It has containers and types and generics. You come to find out that it's all based on meta-programming which is an interesting idea: Basically you're writing code to write code, and that code writes code. The whole system is macros all the way down. There's no native language support for anything but macros and the macros implement everything.
The student's experience is that he can quickly solve a problem using a list of stacks of strings (vector<stack<string> > > in the parlance.) Which is fine, almost like typed-python until you make a mistake. Then, the compiler, who knows nothing about those types which were all built by expanding macros, is not your friend.
Miss a minor `*` and a single line of code will fully expand its underlying macros giving a 10-page, indecipherable error message.
Should you make it to runtime, no debugger can tell you the contents of a vector, string or stack. They're just blobs of buffers and pointers with mangled (and yeah, that's the word that they chose: mangled) names to make them extra unreadable.
This is when most people ask if they can just have C back.
It is entirely possible to write C-style programs in C++ (very few classes, globals all over the place) and so on, and at least for me, C-style software in C++ are much safer/easier to debug than C-style software in C .
You know, it's interesting. I've been programming with Python for about 6 years now. I've also picked up Javascript, SQL, bash, and PHP along the way. I'm always gaining a little bit of C knowledge here and there when writing C extensions for my Python applications. I'm a fairly experienced programmer at this point. To the point:
I tried picking up Go one day because I was hearing so much about how it could replace Python as network glue code with better performance and reliable concurrency. I can't really validate or invalidate those claims. That said, I found Go to be sort of difficult. The syntax is really simple. Compiling is really simple. Concurrency is even simple. However, need to do something in a different way than Go decides is correct? Well, you can't. It won't even compile. The difficulty in Go is in learning about what the compiler thinks is OK. I don't really like that. You don't really know if your code will work until you compile. Basically, I just think Go isn't really flexible enough for modern programming. I find that Nim can do Go's job better than Go can for my use cases anyways.
I think this is similar to what people think about the type system in Haskell or the borrow checker in Rust. With every higher level language comes new things to learn and obey.
This is true in every text-based programming language. s/compile/execute/ for interpreted or repl-based languages.
Will Nim be as versatile and solid as Go in the future? Hard to predict but i would say no. You need a solid financial backing and certain amount of adoption where people actually write software that makes them money.
Agreed, that's why I don't use it for anything super important yet. Nim is approaching a 1.0 release soon. Go has a similar problem in that it is maintained almost entirely by Google who has a history of dropping projects without warning.
> Go is more versatile and at the same time more mature than anything out there.
This is objectively incorrect. In fact, Go aims precisely to be non-versatile for the sake of simplicity. That is why Go does not have generics for instance.
> It is a different design and it excels in what it does (considering all tradeoffs now)
I don't think it's design is all that different. It looks like a stripped down version of C and it's definitely not the first PL focused on concurrency.
> Will Nim be as versatile and solid as Go in the future?
Nim is already leagues above Go in the versatility(I assume you mean flexibility?) department. As far Nim being as "solid" as Go, I'm not entirely sure what you mean. If you're asking about stability, I believe that Nim can reach a similar level of stability as Go, yes.
> You need a solid financial backing and certain amount of adoption where people actually write software that makes them money.
I agree with this. However, it's not always a quick process. The only reason Go is as popular as it is is because of Google's size and reputation(edited). Every programmer on Earth heard of Go within a few days of it's official release. Nim is taking a slow roll approach. Look at Python. It took almost 15 years before it started getting really popular.
All that said, I didn't come into this thread to argue about Go vs Nim. I've been accused of shilling Nim in the past. I'm sorry that I like talking about PLs I enjoy using.
This doesn't seem like a very credible statement on the face of it. Is Go more mature and more versatile than Python? Than Java? Than...Scala? Go does seem to share a lot of use cases (and limitations) with Java; and it would be hard to call it more mature.
It was the same for Linux, Python, Ruby. Being community-driven can be a bug or a feature.
> Go is more versatile
Nim has macros, templates, overloading and compiles to C, JS, Objective-C. Runs on more architectures than Go including arduino microcontrollers.
> mature than anything out there
Go is not more mature than C, C++, Java, Python, Perl...
> You need a solid financial backing
See Linux, Python, Ruby... many projects had no big company or funding behind them.
Pardon me and no offense, but it sounds like you are hitting the 'statically typed language' boundary.. all of the others you mention are fairly loose and dynamic. Go & C, not so much. It sounds like your use of C has been library code, which presumably is more 'data processing' oriented and so doesn't require much structure or control of process/runtime/etc.. which is where you will run into this stuff on the c side..
This is why I moved from c/c++ into dynamic languages to start with.. that said, as I grow more sophisticated and can 'deal' with the typing/lower 'machine' level control, the more I can understand other tools.. even C++! ..
i mention this because each layer of abstraction is there for a reason.. best to view with a fresh pair of eyes imho
You are supposed to "compile on save".
The namespace issues are unfortunately a lot tougher to address.
I get the feeling that the Go runtime doesn't do the same.
Note that you'd run into this bug within any multithreaded process, whether the code was written in go, Java, c, or whatever.
I really don't think this is a language design issue but clearly if you want absolute control you can't get that with the Go run-time.
It's true the Go runtime could give users more control over goroutine and thread scheduling but that would kind of defeat the purpose of not needing to know about it and having Goroutines as the only flexible unit of concurrency.
I think the kludge here is on the Linux side. Having some magic properties bestowed upon threads doesn't make sense. The property should be accessible via a handle that can be shared amongst all threads.
This isn't true in Linux or Windows so I don't see why Go would make this assumption other than poor design.
>The property should be accessible via a handle that can be shared amongst all threads.
It literally is. The namespace information is shared with all threads in a PID group globally available in the /proc fs.
---
What OP is doing is effectively having 1 program run part of itself in 1 container, and another part outside of that container.
Go isn't a systems programming language so this level of fine grain control isn't possible. Hell its pretty difficult in a C/Rust/C++ environment.
