What’s much harder is (1) to expand ones code without coupling everything together (2) present it in a persuasive way that naturally builds a user base.
SystemD feels like the new grad hire who decided the first thing the would do after joining — the opening power play — is to convince the PHB we should rewrite everything in Haskell.
At this point the gentler folk start quietly honing their CVs and sneaking their personal belongings off their desks, one night at a time, in the hope no one notices they are all about to quit.
It felt like Unix systems administration was a haven from this sort of politics — I don’t remember tmux, git, or python trying to push behavior changes on upstream components of the OS[1]. I’m sure it’s why Unix attracted a certain type of personality for so many years and I’m sad to see that changing.
[1] If you are thinking “but those are command line utilities, whereas the init process is a much more specialized case” then I’d encourage you to revisit the Unix principles of every component being a small and simple program! Even init, cron, and login! It’s not some grand ideal to be zealously adhered to: it’s the principles of small components that mean the same OS can run on a quad core Xeon as runs on a $45 network switch.
It was pretty good in the good, old days, where a sysadmin could list every daemon the system runs by heart, but it was breaking apart when you had tons of random packages:
Package X uses "start-stop-daemon" in ini file, so there is no way to get error messages if config file is incorrect. Package B has no disable switch, so you get to stick "exit" in /etc/default. Package C has this weird logging format which cannot be rotated daily, and is not integrated with anything. Package D likes to die and leave PID file around, and then it fails to restart because of that. Package E likes to bury its settings in /etc/init.d/ file, so there are conflicts every time it updates. User A likes to leave their executables on their machine after logout, so we need a cleanup script.
This was seriously getting old -- after you apply a hack after hack, you start wanting something better. There were a few candidates -- upstart, daemontools/runit, systemd. Only two of them actually did the work to add backward compatibility and integrate with upstream. Upstart sucked even worse than systemd. This left only one candidate.
Remember, even if you can claim for Fedora that someone "convinced the PHB", this does not apply to other distros. Debian, Ubuntu, Arch all chose the systemd voluntarily. This is because sysvinit sucks, and no one came up with a better option.
It's as though someone is complaining about having to live under the USSR and how bad things are, and then someone responds that the German rocket program has been dismantled and the British never really had much of one and the Americans are doing okay but the fact is that the USSR was the first to put a man into space. As if that justifies everything else they did.
It's not my experience.
Setuping services has been a pain for 15 years for me. Then with systemd it was suddenly easy.
Detractors keep repeating systemd sucks, but that this point, Debian and Fedora apparently decided it was good enough. And I never heard an end user complain.
I've stopped to be open to convincing.
Actually, it's possible for more than two things to be true at once. It could be:
1. It could be true sysv init sucked. (Actually, it wasn't that bad as a modular version of local.rc, but parallel and delayed startup was always just a kludge).
2. It could be true that systemd is a good init system. (For me it's a step change improvement sysv init for parallel and delayed start up. About my only complaint is you don't see failures on "systemctl start xxx"; you have to run journalctl.)
3. It could be that adding another 1M lines of code into the systemd source unrelated to starting stuff was an unsound engineering decision - almost inexplicable coming as it does from a senior engineer. Keeping things in separate source databases with explicit interfaces is a well understood way of reducing the chances of ending up with a ball of mud.
4. It could be that moving sending state changes via an RPC interface (dbus) so you have a million places you have to inspect to understand the state of the system and how it got there rather than writing the desired state of the system to disk with a journal where everyone can inspect it was a bad idea.
We had a chance to replace sysvinit with something better. Instead we managed to replace it with something that's horrible in an entirely different set of ways.
The most devastating thing I can say about it is that it feels "enterprise." It's unnecessarily obtuse, complex, and clunky. It feels like a port from some other operating system that's been shoehorned into Linux, or like something designed to sell consulting services and training courses by wrapping very simple things in needless complexity.
We could have ended up with something more straightforward, clean, and simple. It's a tragedy.
My understanding (I believe detailed in the article) is that Debian, at least, chose systemd because they were forced to if they wanted a reasonable default GNOME experience for their users. If they hadn't, GNOME would have been missing some features that would have been obvious and jarring.
You can certainly argue that they made the wrong call there, and that they should have pushed back (though arguably by the time Debian got there, that ship had already sailed), but "chose systemd voluntarily" is an incorrect simplification.
This (and much of the article) is incorrect. Debian went through an extensive evaluation process at the time, over the course of 4 months, considering several different init systems, and chose systemd based on this detailed technical evaluation. That evaluation also considered sysvinit, openrc, and upstart. All of that evaluation occurred publicly, and is archived on Debian's mailing lists and bug-tracking system.
Debian did _not_ choose to use Systemd. Debian was forced to choose between adopting Systemd despite going against the very principals of Debian, or loosing Gnome. They were put it a very difficult position.
Ubuntu did _not_ choose to use Systemd. Ubuntu, using Debian upstream, decided not to break from Debian. Shuttleworth, the founder of Ubuntu, has a very sublime message about it at the time.
I'm personally of the opinion that when forced into a corner, do not side with whoever pushed you into that corner. The Debian technical committee did not have that resolve, they took the easy way out. Lots of Debian-derived distros followed suit out of necessity.
PAM was and still is quite hard to understand but, again, it had enough power and flexibility with @ based includes to allow individual distributions to lay things out sensibly and allow new packages to follow a coherent standard that emerged rather than being forced.
Concurrent init systems helped drive a lot of this, back when there was a sweet spot of Linux complexity increasing faster than processing power could handle. (Nowadays I feel like boot speed is less important, given how many more resources a modern machine has, though it’s still a concern for low resource platforms.)
I guess I’ve just never seen revolution as a solution to hacks and untidiness. Linting, guidelines and policy, with technically compelling reasons to encourage people to follow them have seen much more success, where I’ve seen them deployed, than centralized authority.
These improvements weren't real. There was a long tail of scripts that didn't follow the standard and never would. Every distribution still had their own customizations on how sysvinit scripts worked (e.g. debian with start-stop-daemon, suse with insserv, etc.)
And rc was different on almost every major distro family, which made life even more difficult for no particularly good reason.
And there's a certain point where all these things you are doing are ultimately to paper over the "hacks and untidiness" of everything. The introduction of systemd had two major effects:
1. It broke all non-LSB init scripts (because LSB is all it supported)
2. It introduced a straightforward declarative syntax for system resources (units for services, timers, mounts, etc.)
I would argue that the above two things have done more to standardize the Linux plumbing layer and help people take fuller advantage of it than literally all the linting, guidelines, and policywork we did for the decade prior.
If you looked at the discussions that happened during that time (I was there as a Fedora contributor back then, and wow that was a long time ago!), you'll see that we mostly did the change out of weariness of trying to use Upstart features well and failing. And even with that, we delayed it by one release because it wasn't ready during the Fedora 14 development timeframe (it was supposed to be in Fedora 14, we actually launched it in Fedora 15).
When it comes to Upstart, it was almost never used except as a wrapper around initscripts. Effectively you booted from Upstart which greedily synthesized all of its 'events' on startup that in turn triggered initscripts, which brought some degree of parallelism and that was about it. The reverse-dependency 'start/stop on' model was too esoteric, and Upstart's state machine brittle. Hilariously enough, the only system to actually use Upstart to its fullest (still does) is ChromeOS.
Prior to systemd and Upstart, as I discuss in chapter 1, distro developers either simply added various preprocessors for initscripts to introduce dependency headers, or they had these vague hunches of 'waiting for Godot' that we could either replace init entirely with D-Bus, put D-Bus interfaces in initscripts, or some other roundabout way that involved grafting D-Bus. Much of this was owing to the 'irrational exuberance' of Linux developers wanting to hotplug the world after 2003-4, but it always remains an on-and-off effort with the people behind it never certain what they want.
Indeed, besides grunt work of optimizing initscripts, I point out that as late as 2009 distros like Debian were still very concerned about LSB-compliance, and that even when they were willing to adopt a new fancy event-driven init like Upstart, it was conditional on reusing tools like insserv to parse LSB headers, and on incorporating initscripts for service management. Also the entire bizarre diversion that the Debian group had of using a Perl script to generate initscripts.
So far as I know, the daemontools approach was never influential. The available dichotomy was between putting lipstick on a pig, and utopian dreaming.
The paradox remains: how can we trust that the same people who did the Wrong Thing so persistently then suddenly turned out and did the Right Thing like they hit jackpot? If anything, 'jackpot' is the right analogy -- it was a fluke. And the same integration failures that were had with sysvinit have now been shifted into wrestling systemd's job model.
"Unification" was a folly, as the distro is still the place that gets its finger pointed at if upstream can't deliver, and upstream insists it's "just" offering a bunch of "modular tools," which ironically was supposed to be the problem to be addressed! Besides, there is still a whole world of Linux devices out there running Busybox, musl, OpenWrt, etc. that are outside the systemd bubble. Or think of all the container images running Alpine (which is also the basis of postmarketOS). Linux is still the same bazaar it always logically must be, but with a new class distinction added to the mix.
I am also confused why you're complaining about the job model and all its churn and conceptual problems when the whole point is that systemd was offering to handle that for the distros, which is exactly what they did and are continuing to do now. You can claim they are doing a bad job (which is probably true in some sense because of the impossibly large scope of the task) but the previous alternative was that nothing was being done about it at all. Which you did acknowledge, but then you went back to the same criticism as before. Why? It seems obvious to me that the gritty technical aspects and/or perceptions and fears about some kind of false dichotomy are not what could have ever influenced this kind of decision. Nobody else is invested enough in it for that to have happened.
The other reality I've seen is that there is no standard way to actually implement a daemon. Nobody does it in quite the same way and using things like daemon(3) don't help. So you can choose to A. patch all your daemons, or B. you can attempt to simplify the task by writing more tooling. Every single Linux distro I've seen, when given the choice, has chosen option B. So really, I don't see anything here that is surprising or farcical at all.
Of all examples you gave, this one is the least convincing reason to introduce this Behemoth on our systems.
I've seen this scenario played out, and it always fails. The product never ships, of ships in such a bad shape the users have a terrible user experience.
That's not what happened with systemd. It shipped. And after a few iterations, it actually provided a better user experience.
I don't know about sysadmins, but as a simple end user of Ubuntu, systemd is the best user experience I ever had for what matters to me: setuping my services.
It's quite easy to create your unit files, declare service dependencies, stop, restart, and so on. It's declarative, conf is reasonably straighforward to read an understand, documentation is there, and my system boot is stable on laptop and on my servers.
Maybe for "advanced case X", there is problem. Apparently, a lot of power users say there is. But I also know the FOSS community has some people advocating purity over practicality, don't care much about the people with less skills than them, don't understand user friendliness or will fight for their ego. So I have no way to be sure if the problem is real, or just another free software drama.
But at this point I don't care: for the average Joe like me, it's great, and I'm very happy it has been chosen by Centos and Debian as the default.
Apparently nobody in the entire history of init system have managed to provide something decent for, you know, initializing the system. I never heard a herd of devs claiming that one is the fantastic, and they were all terrible to use when I tried.
Well, I don't care for choice in my init system. I want it to work.
And systemd works fine for me.
We Agree.
> And systemd works fine for me.
But it doesn't for me, and I am left with no choice but to suffer with it.
easiest answer is to say get rid of the inter-dependencies, but that's throwing money and time at a problem affecting 1% of users, at no benefit to the rest. Maybe it's still the right answer, but who's to pick up the slack?
The cultures were very different, I straddled them with my hobby working being CP/M->PC->Amiga, and my professional work being RSX-11M->TENEX->TOPS20->UNIX.
Linux (and Minix) started out very much being a PC version of UNIX and PC-BSD jumped in there too. But Linus ceded userland to the community and as the community's membership has grown the people who wanted Linux to be the perfect UNIX lost out to the people who wanted Linux to be an open source version of Microsoft Windows.
It appeared to me that the folks who liked UNIX semantics fled to MacOS, although the complaints on that front are getting louder as well. So here we are.
I don't know why, but it bothers me when someone's trying to criticize systemd, without even bothering to get the case right. It smells of not having looked at it any more than to form the run of the mill opinion on systemd, while looking at what came before with healthy dose of romantic nostalgia.
I don't want my init system to be a series of garage-quality bash scripts with slightly different error rate tolerances etc, systemd solves that. I don't want my system to just blindly shut down, even if it's not necessarily safe to do so immediately, systemd solves that.
What systemd critics often miss is that it's also a rather valuable resource manager in general. I know that many would say this should not be part of systemd, but the thing is while that sounds nice, nobody built anything remotely as convenient for Linux as systemd in this area.
It's nice to say what systemd got wrong, but I haven't seen anything actually built that can seriously compete with it.
I now actively avoid distros without systemd, because the admin experience of those distros has generally been much less consistent.
We can all talk about the right way to do things, but unless there's code that's clearly superior to what systemd offers me today, am not interested.
You know it's written all lowercase, right?
Yes please!
Pure functional languages are very well suited to declarative configuration, especially where you need significant flexibility not afforded by purely static "data" formats such as YAML or JSON.
One problem is that a number of reinventions were poorly made.
E.g. the log format. Yes, unstructured logs have a ton of drawbacks. Can we take some ridiculously well-tested, reliable embedded database or serialization format (like sqlite) and use it as log storage? Alas.
Another problem is the "I know better" attitude.
Are user processes allowed to run after logout? Which DNS to use during startup? What logging in should look like? In each such case, existing behavior which was widely considered as not broken was replaced by a different behavior, often even without an escape hatch. The new behavior(s) in such cases should be made possible, but the default should be the old behavior.
No wonder I don't run systemd on my desktops / laptops.
I don't like the that journald goes backwards in space and updates indices -- I like "append-only" attribute -- but at least it writes data sequentially one after another, which makes it pretty tolerant of data loss.
As for "without escape hatch", I don't remember any functionality like that? In fact, systemd is full of escape hatches which are documented and supported. Just recently, I had to modify "is computer online" check -- it was amazingly easy, all I had to do was to drop-in a fragment file for the right service. I remember doing the same in upstart -- it was a total pain, requiring reading the source code spread over dozens of undocumented files.
What systemd is doing with this log storage system is reinventing sqlite, because systemd has a fairly new and not battle-tested database implementation, it has shortcomings.
I would argue that either they use plain text logs like before or use sqlite but they should not reinvent a new db.
That's great, but you've completely ignored theamk's argument that sqlite is the wrong tool for the job.
Face it: the linux userspace has always sucked. Linus punted on it way-back-when (to be fair, he was a solo dev working on a kernel, he had his hands full) and opted for the gnu tools, but those (other than the compiler and libc and a few other spots like grep) have never been best-in-class. It was always just sort of "good enough" to avoid swimming upstream (no pun intended) and trying to swap them out.
Someone decided to make an attempt at making the situation better in one way, doing things differently. Perhaps it was a mediocre attempt, even. (I'm just speaking for sake of discussion here; I haven't read the systemd code, just making a point.). Still, nobody was forced to use it, and indeed many options for avoiding it exist.
Somehow, this ended in a holy war and corresponding schism. It makes no sense to me. Are linux users upset that someone dared question their decades of suffering under a poorly-designed userspace? Agree or disagree with systemd's decisions, it's hard to fault someone for making and releasing software and offering it as an alternative, even if it sucks.
I'm all for experimentation in this space. The old way of doing things was super janky, and I think systemd is an improvement in a lot of ways. I (thanks to a lot more time stuck in my house) have two recent Gentoo installs, one with systemd and one without. They both work, but one works better.
"This is the way we've always done it" seems in opposition to the hacker ethos, to me. Doing things differently to see if there might be a better way seems like it's always valuable, considering that nobody's deleting /etc/init.d/* from your system if you don't participate.
Traditions are not always inherently valuable. Try new things. Discard the ones that don't work for you.
Which userspace are you claiming is better then? Linux used to offer the best userland experience in the world, because you had control over every part of it, and since things were loosely coupled you could mix and match at will (e.g. prefer BSD grep or Solaris tar? No problem, install them).
> Somehow, this ended in a holy war and corresponding schism. It makes no sense to me. Are linux users upset that someone dared question their decades of suffering under a poorly-designed userspace? Agree or disagree with systemd's decisions, it's hard to fault someone for making and releasing software and offering it as an alternative, even if it sucks.
They didn't build a better alternative and let users choose to adopt it organically. They made a wilfully incompatible, tightly coupled thing, and then pressured other projects to hard-depend on it (e.g. Gnome).
This is quite literally the antithesis of everything that Linux and Unix stands for. The whole reason people were able to use the GNU tools with the Linux kernel is because they are loosely coupled and communicate through standardised interfaces. Systemd by contrast doesn't run anywhere but linux, and you need closely matched versions of everything: your linux has to match your udev has to match your systemd has to match your dbus has to match your gnome has to match your...
If systemd had been around when GNU and Linux were starting out, the whole project would have been literally impossible. It's a betrayal of everything those projects stand for.
GNOME wasn't pressured in depending on it. There deciding to depend on logind over ConsoleKit. Initially logind could be used without systemd, that changed after a systemd version due to cgroups v2. There's quite a bit of overlap between the people maintaining ConsoleKit and the people wanting to make use of logind, so there was a push by some GNOME maintainers to use logind.
The arguments against were often quite weak. When pointed out that ConsoleKit was unmaintained for years the answer could often be summarized as "why does it need development".
> It's a betrayal of everything those projects stand for.
Stuff like this makes maintainers of things just ignore you. There might be a valid point in your argumentation, but why bother.
But that's the trouble. The people who make systemd keep integrating it with everything else they make and vice versa, which makes it harder to try new things because instead of things coming in pieces you might replace, it comes as one big interdependent blob.
I recently installed a Gentoo system with ConsoleKit and KDE Plasma, and it works great, not a trace of systemd anywhere. I also understand the original Gentoo guy forked Gentoo and made a new distribution (Funtoo) where systemd isn't a configurable option even if you want it - it just doesn't exist in that distribution at all, as far as I know. In normal Gentoo you can use OpenRC (default) or systemd.
AFAIK Gentoo also supports Gnome with just logind, too. Haven't tried out that config.
* Systemd was not exactly great when it had to be considered "the" replaecment for SysV init (which does suck), and happened to be much more than an init system;
* The big distros like CentOS and Debian (and then many smaller) had little choice but to adopt systemd, because Gnome had it as a hard dependency.
It's unfortunate that of all SysV init replacements, the evolution chose systemd. By its spirit, it's very anti-Linux: it's often seen as "monolithic" even though it technically isn't, because it tries to control nearly everything, and it does not mind breaking the old ways when it sees fit (and not when the old ways are broken).
Quite likely, we're stuck with it as a mainstream solution for another decade, or two, much as we're stuck with X. (And in two decades, Linux the kernel may become irrelevant, due to hardware and OS research progress.)
Historically Linux users where people that actively choose to use Linux based on a combination of technical & philosophical reasons, thus they are more passionate about it.
Most common Windows users get Windows preinstalled from the store where they bought the computer. It is not really an active choice based on reason, therefore no reason to have a holy war.
The more Linux becomes mainstream, the holy wars will become more & more fringe. Systemd is pushing Linux more into the mainstream.
(FWIW I use nano.)
All those examples you gave are possible to disable in systemd, and in the case of DNS and user process after logout, they are not even the default.
: Killing of user process are enabled by default in upstream but disabled by default in every distro that I know. And systemd-resolved, even if it is build by default, is not enabled by default even in upstream.I also remember that even 2-3 years ago the situation was not as nice yet.
I think what became systemd could have been great software, if not for the, mmm, cavalier attitude of its creators (not necessarily personal; it might be amended by RH corporate deadlines).
And you know what? It’s fucking awesome. Structured logging is just baked in. I don’t have to parse fields back out that have been smashed into a string. I get full timestamps, machine IDs, pid, uid, gid, and all the rest. I get a cursor that I can use to durably resume consumption of logs between restarts. Shit, the cursor even lets me ensure that I’ve consumed logs even when they’re being aggregated from multiple machines, without a single change to my code.
Sure there are a few warts here and there. Everything has them. But it took less than a week for me to be convinced that it’s past time for plaintext logs to die.
Fast forward 10 years and I still cannot find a reliable way to do this with journald, apart from forcing it to output to syslogd and then shipping the syslog logs.
Was is executed well? No, not at all. In typical FOSS fashion it just changed from one release to the other, and that's it.
Yes, there were probably notes about it somewhere in a long forgotten changelog deeplinked 4-5 clicks below the surface of Debian's and Ubuntu's homepage.
Should have they introduced a deprecation period? I don't know, maybe, probably. (But who are "they"? systemd maintainers? Debian/Ubuntu?)
All good software is based on this attitude. Wait, I hear you interject. You have examples to disprove this. Well whatever it is you might think isn't the "I know better" attitude is probably just the same "I know better" attitude with one layer of abstraction.
https://www.reddit.com/r/programming/comments/4ldewx/systemd...
Namely that systemd doesn't allow persistent processes started from the shell by default, preferring to terminate them when the user logs out.
This would include processes like "screen" whose entire raison d'etre is to persist after the user logs out. (Well, it has other uses, but this is the main one IMO.)
The stated workarounds - fiddling with some options like "KillUserProcesses=no" in logind.conf &co. - have so far failed.
I don't know whether this situation is a problem with systemd or the distro, but it seems very much a problem with the culture summarised by the top commenter in the above thread, of (paraphrasing) glibly breaking existing workflows then casually brushing away criticism with arguments often boiling down to: "this is the right way, I don't care about tradition or protecting 'incorrect' usage."
However, if you login to a graphical desktop, that is not a text terminal, and therefore everything is effectively nohup'd: your browser, your chat program, your media player, all the miscellaneous helper processes they spawn, etc. Most people do want that stuff to be automatically shut down when they log out.
Since the kernel's idea of "a login session" is wrong (it only includes text-terminals), and it can't easily be changed for compatibility reasons, systemd implements its own idea of "a login session" that works the way most people (who aren't experts on POSIX job control) expect.
Systemd's revamped notion of "session" includes graphical ones. Yes, that's different from the legacy sense of a mere text-terminal window. But since they've had to reimplement the whole concept of graphical sessions anyway, they could have made them work like textual ones. If you're not running a session manager as part of your windowed environment (almost everyone is these days) the whole thing might break, but then you can just enable the hacky "lingering" mode to make it work and you're no worse off than before.
This is a CADT attitude. Improving existing interfaces while maintaining compatibility is hard; it's also what makes the difference between a serious software professional and an incompetent vandal.
No, it's not.
I think you're looking at this in the wrong way. The concept here is one of sessions and the parent-child relationship between them, as well as the decisions parent processes make when they create and manage (or don't manage) child processes.
If you log in to a text console, you end up with a process (like bash) controlling the terminal. If you run normal programs in the foreground, or even in the background (as long as you don't disconnect them from the controlling terminal), they are all children of that bash process. When you quit bash, all its children get terminated as well.
When you log into a graphical session, the login manager (or whatever) will start your desktop's session (which might just be a plain shell script, maybe just your window manager, or might be a full-blown session manager, or something else entirely). Whatever that is, it starts other applications (say your window manager, a panel or dock, desktop manager, etc.), which then can start other applications (browser, chat, media player). If you log out of your desktop, ultimately what happens is that original session-starter (script, WM, session manager, whatever) quits, and it takes all its children with it.
And regardless, the graphical session still runs in a TTY, just not in text mode!
> Since the kernel's idea of "a login session" is wrong
The kernel has no concept of login sessions at all (it does have the concept of "sessions", but they are unrelated to user login). It just starts something as root (init, as PID 1), and from there userspace takes over and does whatever it wants, including the possibility of starting a getty, which can run login, which can setuid() and launch your shell if you put in the right password. (Or run a display manager that does something analogous with graphical sessions.)
So back to the beginning:
> The trouble is that 'nohup' is not a particular state that specifically marks processes that want to survive logout. Instead, the rule is that processes running inside a particular terminal are killed when that terminal closes, and processes running outside any terminal run as they please.
Those two sentences would seem to be contradictory, no? "nohup" is of course not a particular state, though it is responsible for putting a process into a particular state: that of not having a controlling TTY. Which is (greatly simplified) what determines whether or not a process keeps running once the TTY's controlling process exits.
Help me..
Iron grip, aggression, pushiness are needed to lobby systemd out.
Another important initiative is to not to let Poettering, Sievers and co. continue throwing new systemds onto distributives. A preemptive action is needed.
systemd uses cgroups (kernel namespaces), and if you want something to remain after you log out, then start it in a separate cgroup.
cgroups are control groups, you know, to control processes. Which was simply missing for decades in Linux.
nohup should be enhanced to support systemd, or systemd should provide a nohup wrapper, and just start a new scope (cgroup) for whatever the user launches with nohup.
...
Now, that said, I have no idea why distros and systemd did what they did without much communication, but ... that's usually the Linux way :/
Why should the user be concerned with how systemd happens to implement its session management. If you start a screen or tmux instance, it's unambiguous that you want that part of the session to persist after logoff and to be reachable upon logging on to the system again. That's what screen and tmux were designed to do.
> enable-linger [USER...], disable-linger [USER...]
> Enable/disable user lingering for one or more users. If enabled for a specific user, a user manager is spawned for the user at boot and kept around after logouts. This allows users who are not logged in to run long-running services. Takes one or more user names or numeric UIDs as argument. If no argument is specified, enables/disables lingering for the user of the session of the caller. See also KillUserProcesses= setting in logind.conf(5).
Strange, KillUserProcess=no worked fine for me
How about we give them some benefit of the doubt, that they are mostly trying to make things better, and instead ask them "what is the technically correct way, in the systemd world, of implementing this functionality?"
I suspect the answer, for screen, would probably require some degree of messing with the user manager daemon.
In the case of terminating programs left running in an ssh shell by a logged out user, it's a pretty simple answer IMO: just leave it alone - if the admin of the ssh server wants to terminate user processes on logout I think that's where the burden should lie and there's no good reason for systemd to get involved, that I can see.
I.e.,
systemd-run --scope --user screen[user]@[host]:~$ systemd-run --scope --user screen
Job for run-r0b2a43d19a564e2c9f36b5c9b934420f.scope failed.
See "systemctl status run-r0b2a43d19a564e2c9f36b5c9b934420f.scope" and "journalctl -xe" for details.
[user]@[host]$ journalctl -xe | cat
May 04 03:02:54 [host] systemd[918]: Failed to start /usr/bin/screen.
-- Subject: Unit UNIT has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- Unit UNIT has failed.
--
-- The result is RESULT.
May 04 03:02:54 [host] polkitd(authority=local)[879]:
Unregistered Authentication Agent for unix-process:1504:595444 (system bus name :1.25, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8) (disconnected from bus)
----------------------------
I changed my user and hostname to "[user]" and "[host]" respectively, the rest is verbatim.
It's interesting to note that to even see the error report (such as it is) I'm instructed to invoke a special command rather than just look at logs where I might expect to find it, and that when I do so I am locked into some dysfunctional pager requiring me to pipe through "useless cat" in order to cut and paste the uninformative details on long lines.
While the consistent boorishness of my systemd install continues to amuse, the details of this issue and the fact of its resolvability or otherwise by invoking recondite special systemd functionality are beside the point.
Why should I need to know how to run normal and trusted software in special systemd compatible ways that I've previously had no issues with for decades?
As I opine below, the point is the toxic "you're doing it wrong" mentality that infests the systemd project and its adherents. I've run into this in so many ways since ceding defeat and allowing it onto my systems. This is just the latest in a long line of examples.
Ha, if only that behavior was confined to systemd... I am forced to consider it a defining feature of much of the OSS/Unix community.
Systemd is mild compared to many projects and its adherents in this regard.
It’s just another example of systemd making everything on the system worse.
Top comment on that link:
"""Salient comment: "Or somebody could go find the actual problem @keszybz saw
here - systemd/systemd#3005 - which is: In particular, for my gnome session,
if I log out, without KillUserProcesses=yes I get some processes which are
obviously mistakes. Even if I log in again, I'm much better starting those
again cleanly. fix that, and stop trying to make systemd break the world
because somebody's gnome session doesn't currently exit cleanly."""
> "Shouldn't this code be part of glibc instead of tmux?"
> -- Nicholas Marriott, 2011
> If you want to change how processes daemonize, why don't you change how
> daemon() is implemented [...] ?
> -- Nicholas Marriott, 2016
> * https://news.ycombinator.com/item?id=11798173
> * https://news.ycombinator.com/item?id=11798328
Do you have decent examples of it failing to work? If so, then that sounds like a bug against screen. The link you provide is absolutely huge and from three years ago.
Please give me a simple "stages to reproduce ..." style report. Please keep it simple and short and I'll fill in the blanks if I can and only trouble you for stuff I'm too daft to work out.
remote login, start screen, ctl-A d (detach), log out, log back in, screen -r (resume) - there is no screen session because it was killed by systemd immediately after log out.
(syslog entry: May 3 09:01:25 $HOSTNAME systemd[1]: session-6.scope: Killing process 3290 (screen) with signal SIGTERM. )
But that's hardly the point, the point is the toxic "you're doing it wrong" mentality that infests the systemd project and its adherents. I've run into this in so many ways since ceding defeat and allowing it onto my systems. This is just the latest in a long line of examples.
I agree this is totally unacceptable, and that the inevitable tone deaf response to such complaints is also unacceptable.
I just tried it on a fully up-to-date CentOS 8 box (which uses systemd), and it worked perfectly fine: after logging back in through ssh, `screen -r` restored the screen session as expected.
From what I have read, that screen session would only be killed if I had set KillUserProcesses=yes on /etc/systemd/logind.conf, which is not the default.
The first message in the bugtracker already has the context of whats happening.
Poettering:
> Note that in some cases it might be a good thing to kill screen sessions when the user otherwise logs out […]
* https://lists.freedesktop.org/archives/systemd-devel/2011-Ju...
For the lazy, here's the context of that cherry-picked sentence:
> In order not to break screen we currently do not set kill-user=1 or kill-session=1.
> Note that in some cases it might be a good thing to kill screen sessions when the user otherwise logs out (think university, where students not logged in on a workstation should not be able to waste CPU), in other cases its a bad thing however (i.e. in yours). That means it must be configurable whether screen is considered an independent session or not.
> One thing I’m certain of is that this shift cannot emerge from dilettantes, outsiders and proverbial basement hackers. One does not unseat a platform without already being part of the patriciate that calls the shots on what gets integrated where across the largest nodes in the ecosystem.
It’s a long, turgid “why wasn’t I consulted?” complaint which really just comes back to the question of how open-source projects work. An init system is harder than it might seem at first and requires buy-in from an unusually large number of parties since it affects the OS, everyone shipping daemons, and the operators. If you’re not going to invest that level of engineering time, I don’t see how it’s reasonable to expect an equal voting share with those who do.
systemd-as-init-system is not the problem. systemd-as-kitchen-sink is the problem.
It's the tight coupling that annoys many people.
Does udevd really need to be in the same repo? While there may be some nice things about journald, does it really have to be in the same source package? (And why can't it support remote logging with the industry standard syslog protocol? Now I have to run journald and rsyslog. And why doesn't it have an ACID file format that doesn't self-corrupt at times? Why couldn't they just use SQLite or OpenLDAP's LMDB?)
What does systemd 245's systemd-homed have to do with an init replacement?
* https://www.techrepublic.com/article/linux-home-directory-ma...
10 years ago this would have been considered satire
No one is forcing you to change your broken buggy 30-year old shell scripts, you can always continue using them.
I like the overall theoretical concept of systemd but it has ugly implementation details like this.
Unfortunately, currently systemd-resolved is the only OS level DNS-Over-TLS solution that works for my use cases: roaming laptops and desktops running Linux. At home it's fine because the core router runs AsusWRT-Merlin which does DNS over TLS for all clients using it as resolver.
I don't have a strong opinion against systemd (initially I disliked it moving from Arch's BSD style rc.d). Trend is that systemd is taking over more and more *ctl, seem like its goal is the entire user-space?. Most distros choose to accept and adopt it, and overall (at some point) it eliminated distro differences and lowered the barrier for new Linux users (desktop) and sysadmins.
Personally I started systemd when it rolled into Arch Linux, after which point, rolling upgrade has never gone broke (workstation use case - Desktop Environment / WM), could be skill levelled up after gaining experience though lol...
BTW: I watched `reinventing home directories` by Lennart Poettering (recording @ All System Go! 2019), interesting one. Personally I like to see reinventing wheels to solve legacy problems (people get used to them, sometimes confused, question but don't know how to solve) others don't even consider problems. These helped open my mind and look at things from a different PoV.
[1]: notes on systemd-homed https://sites.google.com/site/imterry/aboutme/profiles/skill...
udev doesn't have to be there, but when a handful of people maintain critical parts of the Linux ecosystem and they saw ways to make their maintenance work more effective, they went ahead and made their work more efficient.
I mean, criticizing systemd/udev/Debian/Ubuntu/Arch on how they did stuff, and the technical merits of how things are is completely fair, but criticizing "why" is strange. They did because they saw that it will help them, their projects, their users. (And they probably understood very well that it won't please all of their users.)
An init system is harder then it seems ? You and Jonnax seem to think that unless you made it you have no right to say anything. Have you ever made an init system ? Researched what exactly goes into it ? Sure, there's more then just starting programs. But not much more. The only really complicated thing about getting a [linux] computer from kernel boot to firefox is setting up devices. And even that [udev,"hotplug"] is not complicated as much as it is.. obscure. I know about these stuff, do you ?
I'l defend a newbies right to say anything. Nobody was born with all the knowledge of the world. I will not have compassion for your kind of ignorance.
SystemD is about as much an init as my foot is me. That is the problem that most of those you so casually dismiss have with it. That and that it wants to become mandatory. Could they have just extended udev instead of integrating it ? An i wrong to say that ? Is my opinion wrong because i don't have "Red Hat senior engineer" next to my name ? Programming is just a hobby for me. I don't even have a degree in it. So i'm ignorant on the subject ?
Funny thing.. most of those on teh internets who are vehement in favor of systemD don't know much about how a computer system boots and what goes into that.
edit: Never mind. I see feelings and fancy words are still more valuable then knowledge or thinking. Maybe one day, when computing is not as young of a science as it is now.
I know enough to know that you left out networking and authentication, network service dependencies, filesystems, volume encryption, etc. – not to mention automatic restarts, conditional behavior based on the network or other devices (stuff needs to happen when I leave the office), and the various edge cases which I’ve seen in three decades running Unix systems. Every other operating system Linux is competing with has a much better model than SysV init (Windows has since the early 1990s!), and the world changed from static servers which rarely changed to most systems needing dynamic behavior – what worked in an old server room isn’t a good fit for a laptop!
Again, I’m not saying that systemd is perfect but that most of the complaints tend not to acknowledge the many problems it solved and a decade of angry forum threads doesn’t seem to have resulted in any concerted effort to build a serious competitor. The Debian community had a deep examination of the alternatives and picked systemd because it made sense, not because Red Hat has some brainwashing program.
Arch Linux switched to systemd earlier than Debian. And I have not noticed. Arch community accepts breaking changes. Debian? I believe they value stability.
Systemd should have matured, proved its citizenship and qualities. And it would have been accepted. Instead it usurped. People had no choice but to abandon their platforms. Decade of angry forum thread shows that there is a problem. Just like with python3, it's better to learn. And it is especially painful that there was PulseAudio just before. Same pointless change. Removed it and never looked back. ALSA works for me, still.
I’ve been running Unix systems for 25 years. All of the things you mentioned were already solved adequately in the Linux 0.9 days, and the BSD’s still use old fashioned shell based inits, and work fine in my experience.
In fact, the only time I’ve had issues with any of this stuff has been in the last ten years, and only on systemd-based distros.
The genius of init systems is that they factor all this stuff out, and just handle starting and stopping daemons.
You say i left out stuff. I did. A lot more stuff then you wrote here. It would be a looooooong list of things, all of them simple (including "networking" and encrypted filesystems). I mentioned udev as an example of something slightly more complex, but even it is not.
> .. much better model than SysV init ..
You talk as a user, as in an admin. I talk as someone who was once interested in how all these stuff actually work. What i played with (read about, coded, and researched in general) includes init, udev, network hardware management (how ip and iw actually do stuff, excluding wpa), and more. What have i learned ? That almost all of it is done by the kernel. SystemD is not the only.. program that responds to events and goes over a graph and such. In fact, it is not even that hard to do what it does. Remember how i asked why udev doesn't have a generic event.. protocol ? You can scrap udev if you don't use systemD. And why ? For better integration ? The kernel tells udev when a device appears. So why doesn't udev just send a message of some sort ? It could even have a "mode" where it doesn't do anything unless it receives a message telling it to. So why not ? Oh, i don't think it's malice.
> The Debian community had a deep examination of the alternatives and picked systemd because it made sense, not because Red Hat has some brainwashing program.
The debian debate was a clusterfuck of grand proportions. There was one (1!) post on the whole debate that addressed the technical aspects. In the end the conclusion was that it is easier to just use systemD as stuff depends on it more and more (GNOME, notably, funny). RH is a huge company, they are not altruistic in their goals. I wouldn't say malice, but i wouldn't say innocence either.
And again, nothing of substance was said. Just fancy words and feelings. Brainwashing..
PS Fun fact: Xorg depends on udev to send it events over dbus when you plug something related to it. It still works without it, until you re-plug your mouse.
It solved nothing but the problems made up to justify its existence. And the ones those faux solutions created downstream. Boot ordering problem? Feh. What a gaslight fest. All caused by chasing 1.2s faster boots. What a colossal waste of everyone's time. This is the same cancer that Shuttleworth tried to foist on everyone with that UI abomination. Servers are desktops are laptops are tablets are phones!
Why can't you have a discussion without childish insults?
Linux users spoke and set the system up to do exactly what it does--use a dominant position to drive out alternatives. They only complain when they suddenly wind up on the receiving end.
Perhaps if you don't like getting the pointy end of the stick, maybe you shouldn't sharpen it in the first place?
I've read it more as a quirky history writup with some technical and idiology discussions.
FWIW I still think the technical criticism in the article has a lot of merit and what seems "turgid" to you is a nice link collection to learn about systemd minutiae...
Sometimes vocal minority is just that.
they work differently depending on the project.
so this statement:
> requires buy-in from an unusually large number of parties since it affects the OS, everyone shipping daemons, and the operators.
is only true in some cases.
c.f. rcng, which was developed by a very small few people, and is a generally sane improvement on previous BSD init that i've really never heard any complaints against vs the previous method.
same could have been true here. it was not - and the need for 'consensus' was largely sidestepped by the 'market pressure' of gnome being dependent on systemd features, and the leading core desktop environment that people had already attached themselves to, esp in the enterprise market.
This statement makes no sense. It is well understood and has been studied and critiqued by many, including me. Also, I don't know what "social level" has to do with anything here. It was just created for the commercial aspect of Linux and forced into the relevant distro's by commercial interests... it's just that simple.
But it was made by guy hired by Red Hat so instead of maturing and having to take actual user feedback before being used in anything significant it was pushed by a guy that vision is limited to "It works fine on my laptop".
Who then decided to reinvent everything along the way like log storage that still takes over hundred of file opens (and tens of seconds) to get "the last 5 log lines of a command" ( https://github.com/systemd/systemd/issues/2460 ), because why the fuck use say SQLite when you can have fun evening inventing your own shitty binary journal format.
I do find a lot of stuff behind systemd useful but the ever-present half-assness of its solutions is annoying. Half-assed logging here, half-assed dhcp client there, quarter-assed ntp client elsewhere, all while basic functionality is still riddled with edge cases.
Still a net positive over having to fix yet another fucked up init script...
Really, the issue here is that "how it used to work" is mostly unambiguous, but I have no clue what you mean by "the spec", and I suspect you're actually referring to a multitude of specs, several of which were only created in the process of writing the relevant systemd components.
I thought the reasoning was to make logs harder (or impossible) to tamper with.
Something something about trading freedom for security.
the binary logs handled by the systemd journal can be "sealed" at regular time intervals. That seal is a cryptographic operation on the log data such that any tampering prior to the seal can be detected.
This might be news to some people.
I remember reading some advice about not doing rm -rf / on UEFI systems and looking that up, it looked like some MSI laptops would brick if certain UEFI variables were wiped.
I found an issue in systemd's issue tracker. And from what I could tell the ability to write to these variables was required to perform certain functions like rebooting into the UEFI system.
But it looked like that some forum or another had linked to the issue. And plenty of people were up in arms about it without anything substantial to say.
If we're going on technical credentials, I have a few minor ones. I'm not writing dozens and dozens of lines of C every day, and I'm definitely not writing kernel components- but I've been doing low level technology stuff for 15 years now.
Everything from watching access patterns on drives and recombining data to writing/patching memory allocators and aggressively tuning system internals. I'm currently responsible for automating the deployment of the largest always online game my publisher has ever known. I'm relatively technical and highly regarded for my technical competence.
When it comes to systemd; you could argue that I'm "stuck in my ways"; but I've used systemd since fedora 21, I saw bugs you couldn't imagine, I read the code, I saw the hostility of the developers (maybe it's just defensiveness?) and I was a little worried, but ok, it's just fedora, it'll be better in RedHat when it releases... and while a lot of the lower hanging bugs went away for most use-cases; many remained in the edges. Instead of them being fixed though there's been a propensity to build a software eco-system and "system layer" to the OS.
My argument has never really been that "systemd totally sucks" it's that the conversation surrounding it is very hostile.. If you say that you don't like journald (because it's very easy to corrupt and has access drawbacks, encoding issues, tooling is sub-par etc) then people shout at you: "YOU CAN USE OTHERS", but you can never turn journald off.. it's wasted IOPs, IOPs are not free.
The biggest issues I had with systemd is the fact that (unlike Solaris' SMF, which is great btw) it will hide failure from you, and failure can happen relatively randomly because nothing is deterministic. This is not a holdover from being multi-threaded, there is actually a dependency graph that gets generated. But I've never been able to access that. (Maybe that's changed?)
My argument has always been, instead: "systemd's tight coupling of so many components means that we will never be able to have an alternative or successor to systemd". Because re-implementing single components is next to impossible and the sheer scope of the "system layer" is many times more lines of code than the kernel itself.
Anyway the conversation is steeped in some level of tribalism.. Which I truly hate.
You can't talk about systemd without being "pro" or "against" and then people take extreme stances, like "systemd saved my baby" and "sysvinit was trash" (and on my side: "unix philosophy!", and "systemd ate my utility!")
They had the people because they had the money. The debate about systemd is only partly technical; it's mostly a culture war about how Red Hat is hostilely taking over the free software ecosystem, and how that invokes memories of embrace/extend/extinguish.
It seems a lot of people simply don’t want to understand systemd, and go out of their way to avoid learning anything about it.
Could that be considered a social factor?
I hope you're being deeply ironic.
It's several orders of magnitude more complex. It's more difficult to understand and reason about because its internal state is a black box, and the number of interfaces and file formats to understand is again orders of magnitude more complex.
sysvinit had a socket interface with a single message type. It had a single and very simple configuration file (inittab). Everything else was delegated to higher levels. Simple, flexible, and extensible. But above all, understandable in its entirety, and completely predictable.
As a user I 100% disagree. Before systemd I had to learn a few dozen of different config formats for various things * various distros, now all my systems are configured in a reliable & simple way, for init, network, timers, etc etc
Read: A confusing mess of different daemons, scripts, and configuration files, which neither worked together elegantly, were simple to configure, nor were easy to debug. Before systemd I would avoid writing custom daemon configurations at all costs because when I did I would need to learn how this version of this distro did it, fiddle with the byzantine mess of scripts, and inevitibly debug multiple subtle issues, now it's basically trivial to get a new daemon running, and I extremely rarely need to debug issues (And I have had to track down systemd bugs due to obscure use cases like one systemd unit modifying other systemd unit configs at the same time another systemd unit with multiple execstart lines is running).
I’d like to setup a service, which depends on another service, and which must always be running, and if it goes down, it needs to have all forked processes killed, must be restarted, and it needs to run as a specific user.
With systemd? 5 lines or so of boilerplate, independent of distro.
I don’t know about you, but I call that simple.
On systemd, it is "systemctl mask NAME". On sysvinit? Well, there maybe a file in /etc/default/ -- but each package has its own option name... and some of them cannot be disabled at all.
it is so easy to deprecate someone else's skills and knowledge with massive change like this. would it be that hard to respect some of the users?
I think there's a lot of people who get caught up in the merits of systemd and assume that there's nothing better. Or that the alternative is the old sysvinit which was in dire need of being replaced- but that's not true.
Process supervision is totally feasible in init (just look at runit, Solaris' SMF and MacOS's launchd) even Windows has one.
> As it turns out, there was a little-known Freedesktop-affiliated project called xdg-hostname in 2009 which was a start towards creating such D-Bus “utility daemons” for use in desktop widgets and others, all as a standalone project. Had this been followed through instead of having the utility daemons end up being part of the systemd source tree, a good deal of political acrimony could have been avoided – though at the cost of reducing systemd’s leverage.
That is, if they used xdg-hostname instead, SystemD wouldn't be tightly coupled with Gnome, so users could choose between SystemD and InitV, and all this mess could have been avoided.
It’s as if web developers were told now that there is IE, that is prevalent, we can write web apps that only work there ;-)
Can’t be impossible to distill some basics about service startup configuration and dependencies, can it?
Then make it extensible and cry about differences in extension support like we do about per-browser css options, but at least converge on some basics.
If we're talking about init; there are actually better inits already available but you'll never hear about them (runit, for instance) precisely because you don't just leave systemd.
You leave the systemd ecosystem.
This is the main point of the article: systemd as a "job system" is actually quite bad, but the ecosystem has some value in some places and you have to buy the shit with the pig.
I do now. Regressions are a bad thing.
Not for long, probably. I drifted out of this sphere years ago, and came back specifically for systemd's 10 year anniversary, as I felt obligated to at least do that.
Judging by the tone of the comments, 10 years later is still too soon to discuss systemd dispassionately, but when I come back in 10 more years I'll see if things have changed.
Regardless, I did have at least some expectation that people would actually focus on the original post and not brawl in the comment section over tangents that often aren't even part of it.
Oh, well. And hey, if other people down the road read this and it makes something click for them, I'll be happy with that.
The usual arguments are it is not "Unixy" and then fall back on original designer intent and philosophical attacks, or worse ad hominems against Red Hat or Linus Pottering.
Your average HN user doesn't see or particularly care that it makes things easier for distro maintainers, so it's much easier to push the corporate conspiracy angle than to accept that a lot of the hand-wringing takes huge amounts not-very-fun volunteer labor for granted.
> Indeed, one ought to observe that the most important consumers of an init system are not sysadmins and ops people as commonly believed, but rather distro maintainers. It is their laziness that determines the balance.
Many people have compared free software to communism. I always used to dismiss it. But, in a way, they were right. Both are movements with a self-conceited historicist endeavor to eliminate exploiting classes ('it is inevitable that the tendency for the rate of profit to fall to produce immiseration as capitalists can no longer profitably invest, leading to a proletarian seizure of power' versus 'it is inevitable that proprietary walled garden development models stagnate as they're overtaken by decentralized communities pooling their knowledge and labor into the most optimal end' and create a perfectly equitable free association of producers no longer having their surplus value be seized by exploiters.
One degenerates into a repressive bureaucratic collectivist (I'm not a Trotskyist, but it's a good term) mechanism for reproducing surplus through either direct requisition or coercive planning. The other degenerates into a way for large-scale cloud providers to appropriate the free labor of volunteers so as to reinvest it back in their walled gardens.
Stalin represented the Leninist wing against ultra-leftists, anarchists and other factions who were against tight top-down organization in favor of 'spontaneous' organizing, and the GNOME/Freedesktop/Red Hat nexus is against the faction in free software that insists on a loosely coordinated and loosely coupled bazaar without a central vision. But in no sense has systemd abolished this, it has only reduced a few nodes at best.
EDIT: Actually I should point out that 'free software' per se is a deontological argument, and it's 'open source' specifically which specifically uses these historicist arguments to justify itself. But then open source was unabashedly an effort for corporate sanitization of free software, which most people forgot after it mostly crowded out 'free software.'
Of course it would take some time to migrate existing unit files.
It doesn't sound like a fundamental critique. Would be nice to turn this into a constructive proposal to fix systemd.
This is a really entertaining writeup, given the subject matter.