But I just have to ask, even if it sounds condescending (I'm sorry- but I really am just honestly baffled-) has anyone who complains about systemd actually written a unit file vs an init script?
I have used init scripts, I have written init scripts. I hated it. Every program needed it's own little way of doing things and writing an init script for a daemon was a hassle of bash scripting and copy pasting.
Now you can get so much functionality out of literally just an ExecStart with a few lines of boilerplate. It's beautiful. And all your logs are in journald, which for me get scraped by a single promtail config into Loki. And systemd's structured logging means I get a lot of metadata available by default for filtering and tagging.
It's just swell. And it's free software!
/rant
They made binary format THAT DON'T HAVE ANY FUCKING PROPER INDEXING. It might have actual indexing, who the fuck knows, but it don't have any useful one
They could literally just drop in SQLite there and now your entire's system's logs have power of being queried via SQL which would be amazing and easy to use with countless tools that can be used with SQLite but NO, (probably) Lennart wanted to have some fun writing shitty binary format coz of his chronic NIH.
strace systemctl status 2>&1 |grep /var/log/journal |wc -l
71
systemctl status tinc
journalctl -a -e -n10 -u tinc |cat
-- No entries --
I can literally grep a bunch of text files faster than it can look thru it's "efficient" binary logs. But hey if you waste RAM to cache every journal file on your machine in RAM instead of, dunno, actually useful stuff, those commands are fast...
It's an abomination and affront to good engineering. Which is shame coz I like many of the other parts.
/endrant
Systemd is 13 years old.
As far as I'm concerned, the complains against churn run out after the project is old enough to drink. Especially if it's not just well adopted, but by far the most popular option.
In general, having open source but being afraid to change it sounds like hell to me. I personally believe in Linux & open source because it lets us keep improving & iterating & changing. Open source embodies the soft in software, and is the best vector computing society has for bringing light & clarity to a confusing but powerful man-made microverse.
I still don't like it.
The binary interface to logs is a good example, not only for the popular dislike for needing to run a binary to view the log, but also, central to my dislike, is the issue with the "init" system being in charge of logging at all.
systemd if far far beyond an "init" system, it is striving to be the "kernel" of user space, in charge of all system functions that are not in the kernel.
This isn't feature creep, it's by design. Is the "init" system in charge of network name resolution? What about making fundamental configuration setting to the OS? Another example of the binary required to access logs, is any change to a non-running root filesystem. If you've configured a rootfs for a machine other than the one you're doing this work on (maybe a different CPU architecture, or simply a rootfs for another machine) you're going to run into complications. Same situation when in a chroot. If the rootfs you're configuring, isn't the one currently active, those binary executables are going to havee trouble.
As soon as we have systemd-packagemanagerd, there will be 0 difference between systemd based distributions. Everyone will simply put the kernel together w/ systemd for a complete operating system. This was Red Hat's intention, now IBM's intention.
I use linux for my work, both in embedded targets and in my development workstation. For many years I've simply copied my home dir to new computers and moved on. The next time I configure a workstation from scratch, I'm going to voidlinux. runit, based on The DJB Way, is much closer to my ideal...
over which part of linux's history do you feel this happened?
during the years long libc migrations? during the thread wars?
which linux-based OS have you used for a couple of decades that hasn't had large changes but has managed to continue to exist? slackware?
We had, no joking, thousands lines of init scripts that were just ripped out of distro (mostly CentOS/RHEL... Debian had some pre-made wrappers that mostly did the right thing) and fixed by us because people saying "it's so easy" to write init script ARE LYING.
And some apps, like Pacemaker (app for making other things HA), are really sensitive on scripts not being done fully correctly
Some tidbits:
* script reports app is stopped right after you run start... because it left creating PID to application, app took some time to boot (Java), and the script didn't wait for PID creation. So running "start, then run status to check whether it worked" returned that app did not work.
* script reports app is running after reboot. The app is not running. Script read off PID file off /var/run (that at the time wasn't in tmpfs like now /run is), saw PID exists and didn't try to start it at all. The pid was of a different application. That was MySQL, not some random obscure app...
* "stop" action doesn't actually return when app is stopped, it just sends signal. sysv goes ahead and shuts down system before app manages to write all of the data. Or in case of pacemaker it does weird stuff because "status returning running after sending stop" should not happen.
* other various ways "status" was broken randomly
That kind of thing leaves a bad taste in some peoples mouth. I am pretty sure some people have bricked hardware with systemd.
* https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=825941
As someone who runs mostly servers, I also find resolved annoying and have to rip it out. There's also all sorts of tomfoolery with /etc/resolv.conf in many modern distros, to the point I've taken to setting the immutable flag on it to prevent things diddling with it.
There was also the bug where systemD wouldn't allow usernames that started with a number. Poettering's (initial?) response was to tag it as "not-a-bug":
> Yes, as you found out "0day" is not a valid username. I wonder which tool permitted you to create it in the first place. Note that not permitting numeric first characters is done on purpose: to avoid ambiguities between numeric UID and textual user names.
* https://github.com/systemd/systemd/issues/6237
* https://news.ycombinator.com/item?id=14681377 (discussion at the time)
Nevermind that (a) POSIX allows it, and (b) so do the user manipulation tools of Poettering's employer at the time (RH).
You may disagree with the architecture or design decisions made with systemd, but I don't see how it's possible to argue that SysVinit was in any way actually "better".
Systemd does establish easy-to-use & incredibly-powerful access a wide range of super-powerful knobs. And in a consistent & clear way, where-as each init-script would have a totally different way of doing config & slightly different approaches to the various hacks. And init scripts arent composable, where-as you can take your distro's regular unit, and drop-in some additional configuration easily, in a very eloquent way, across any bit of config. I think systemd is amazing & brings enormous clarity. It has enormous power.
But init-scripts still had more power. But it was reckless chaotic power that was enormously hard to tap, and required prodigal talent to develop these individual artisinal artefacts, which became, as a user, a thing of horror to rework. Init-scripts almost never were particularly flexible or powerful, but so help you if you ran into such an artefact.
Re large attack surface, anything prior was just security by obscurity, if not worse. You can’t secure chaos.
Out of the box, and can be used with every single systemd unit file.
Isn't that amazing?
Linux has never really had that culture of finding the most concise but most general solution to a given problem. The results of that are everywhere, but systemd is a prime example.
To be fair, the world is not as simple as it was when Unix was designed either, and that's only partially because the Unix philosophy hasn't been widely adopted.
Then they should have consulted some CS books, as it doesn’t compose in the general case.
Also, how was using another init systems command different than using systemd’s?
He also fucked up some in systemd, journalctl being biggest example.
He also fucked up some in kernel, to the point Linus yelled at him
The critique is entirely deserved
I don’t use sysv init anymore but I don’t feel like I ever fully understood those scripts.
Don't get me wrong, I've made my peace with systemd and I've never been an outright hater, but I did more sysadmin work in sysv and the transition times than now but I still get bitten by systemd more often per year. Usually small stuff, but apparently it only made those things better that never bothered me.
Yes, I've written quite a lot of both. My objection to systemd isn't really the init system replacement. It's OK (although I really don't prefer it over the System V style). My real objection to it is everything else about it.
> And all your logs are in journald
JournalD is an example of one of the "everything else" that I strongly dislike.
Some people just don't like it. That's their decision. In most cases its emotional.
Beyond complaints about journals being binary, which I get, there's usually not much more to the systemd hatred than "I just don't like it, humph (frown, arms crossed)".
It's an open source world..... the haters have the chance to come up with something that's better and if it is actually better then in the Linux merit driven ecosystem, then the better thing should win and be adopted, right?
Personally I love systemd - unbelievably useful and powerful.
Stallman used to say Linux should be called gnu/Linux because there's some gnu code in many distros.
More realistically it should be called Linux/systemd because systemd defines so much of how the operating system works.
I have a prototype FUSE filesystem which renders the journal as a tree of .log files (https://github.com/awfulcooking/journalfs)
But my muscles finally learned `journalctl` and after reading the docs I mainly use it.
Patches welcome to make journalfs practical (i think it may be lagging output when you `tail -f` atm)
Would love to see systemd offer a virtual filesystem interface by default.
But seriously, I expect to see a lot of software lose the ability to be run on systemd because they don't ship a service file.
A systemd unit is a simple thing to rustle up. I have recently been informed that an OpenRC script is too these days and I'll fiddle up a VM and have another bash with Gentoo. My job as Managing Director of my company is to maintain an open mind!
I don't think that not shipping a systemd unit will crap out any piece of software. I've seen quite a few packages of code, source and that and an initscript in ./contrib probably works on Debian and farts loudly on SuSE or vv. Who cares? systemd can run it quite easily and with minimal pissing around. With logging and all the other useful stuff.
(Generators are arbitrary programs that generate systemd units at runtime dynamically. Eg fstab-generator parses /etc/fstab and generates .mount units. The sysv-generator generates a .service unit for each sysv init script it finds, and is what is being removed here. https://github.com/systemd/systemd/blob/main/src/sysv-genera... )
This is so much nothing it can’t get nothinger.
But actually taking those words at face value...in what sense could systemd be described as "embracing" or "extending" any init system that came before it? Wasn't it always a from-the-ground-up different thing?
Poettering has been a Microsoft employee for like a year now.
How's that "extinguising" anything?
That said, there are a lot of legitimate grievances out there against Systemd. But it has little to do with using unit files instead of scripts.
Would it really be that hard to have a symlink from /etc/init.d/apache2 to /bin/systemctl and teach systemctl to do the right thing?
Why are you making it sound like a monoculture is a good thing?
Not really, it has increased the number of people using alternative init systems like OpenRC, Runit, S6, etc.
Before SystemD, SysV was even more mainstream than SystemD is today.
is he still working on systemd?
https://github.com/systemd/systemd/commit/7474097d51907c5cc1...
Good luck out there, friends.
The more impactful change to me is the one mentioned a few lines down, about `systemctl --user` getting `CAP_WAKE_ALARM` so that user-privilege programs can set timers to wake up from suspend. As it mentions, GNOME Clocks has wanted to use systemd timers for a long time, and it's the de facto alarm program for Linux phones running GNOME / Phosh. https://gitlab.gnome.org/GNOME/gnome-clocks/-/merge_requests...
-rwxr-xr-x 1 root root 52400 Apr 3 02:25 /sbin/init
$ cat /etc/debian_version
12.0
running:
postgresql, postfix, dovecot, clamav, nginx, chrony, clamav, isc-dhcpd, bind...
RootEphemeral=, which lets processes run in a filesystem snapshot; super neat for seeing what changes a process might make to your system. https://github.com/binpash/try was a popular submission a couple days ago, & this builds in a similar capability to the os.
RestartSteps= and RestartMaxDelaySec= to finally finally offer some level of control over exponential-backoff in the restart behavior.
FileDescriptorStorePreserve= is kind of mysterious, lets systemd store open file descriptors for a process even after it terminates. Some interaction with systemd-analyze. Not sure what this is for exactly but definitely cool extra visibility/observability of processes. I tend to think it also potentially could be useful allow for some weird/fun stuff like restarting services without closing ports.
Soft-reboot which can be useful for switching roots without having to juggle bootloading. Very useful for anyone doing partitioning or what-not. And as I hinted above, you can pass file-descriptors, meaning you can avoid service-interruption while rebooting; what a neat trick.
DelegateSubgroup= puts processes inside a sub-cgroup inside the top level cgroup. So as a user you could have a bunch of sub-cgroups that you can manage, letting you juggle/manage resources within your cgroup. This sounds a lot like Dixon's incredible talk technique of cgroup management, Linux memory management at scale. https://www.youtube.com/watch?v=beefUhRH5lU&t=2241s This is also easily what I think the worst most pitiful saddest part of the Kubernetes story is: a complete unwillingness to let Linux manage memory & use cgroups well, in favor of it's own scheduler which makes all the decisions.
iocost calculations for io devices, to configure/tune io/qos cost for devices. has a huge existing db of drives. https://github.com/iocost-benchmark/iocost-benchmarks
systemd-sysext gains a systemd-confext twin. the former manages overlays for /usr and /opt, the latter manages overlays for /etc. Very nice portable services tool now. Maybe the capstone work completing Revisiting how we put together Linux Systems. https://0pointer.net/blog/revisiting-how-we-put-together-lin... .
Using efi vars to save the hibernation partition, which should be a huge win for hibernation working everywhere.
Some other neat bits:
CAP_WAKE_ALARM rights to user sessions, for setting long-running timers. Sigqueue support to send processes signals with associate values. list-paths verb to show paths. systemd.tty.* kernel arguments for setting up ttys. Early load virtio_console module if it detects it's runnin in a VM to get console up earlier. Upholds= gets a new .upholds/ drop-in directory for keeping other services running (not just starting them like Wants=). sd_journal_get_seqnum api call so apps can have mechanistic sympathy with journald. Ton of bootloader, boot-security, repartitioning, disk-encryption enhancements across the map. SetTTY to for updating the TTY of a session; so useful for ssh'ing in.
udev creates /dev/loop/by-ref/* files with helpful names. systemd-resolved gets a StateRetentionSec= to let you cache & use old records if a nameserver isn't responding. Most systemd services now have some standard signals they accept to tweak config, ex: changing log level, log target, trim memory, print debug mem info. machinectl & networkctl gain "edit" and "cat" capabilities for viewing config. systemd-firstboot --reset which is hella useful for cloning machines & updating machine-id. Services now sd_notify with their EXIT_STATUS on exit; part of trend of making services more observable/operable everywhere. new systemd.mount-extra= kernel arg to ask for mounts at startup.
I love reading release notes in general, but this was absurdly delightful. It felt like in the past sometimes systemd was focusing on new feature growth & ideas, but I'm seeing huge maturation & growing in in this release. Really leaning hard into everything alive & responsive on via signals and passing around file-descriptors is excellent excellent excellent systems-wonkery. The fit and finish here is going way up. There's just a ton of damned-useful ergonomic wins that make everything super easy to find & see & touch & manage. 254 is an epic release.