And yet I do the _exact_ same thing with systemd. I don't get it, I don't like it, and every time I need to do _anything_, I'm googling, copy pasting commands into my terminal until something vaguely correct happens. The CLI options are indecipherable to me, like reading hieroglyphics.
Deep down I understand that it solves some pretty complex things when it comes to system initialization, but something about the UX of using journalctl and systemctl is so bad (for me) that it creates a mental block that I cannot overcome.
Luckily, nothing I do in my day job requires messing with systemd.
https://www.freedesktop.org/software/systemd/man/latest/syst...
It's pretty well documented, especially for Linux. Follow the links from that page at least one level deep.
Part of the complexity is that it's tightly integrated into the same Linux concepts that enable containers. In fact, systemd-nspawn can manage containers on your behalf.
But start with the concepts and unit files, then read the man pages for systemctl and journalctl. That'll get you most of the knowledge you need to interact with it on a day-to-day basis.
You used git as a (counter?) example, but for me systemd has this issue because it's because I touch it so infrequently that whatever I learned about it has aged out of mental cache.
I have the same issue with `jq`, GNU `parallel`, and `ffmpeg` (which I'll grant even if I used it daily I probably wouldn't be able to keep up.
> UX of using journalctl and systemctl is so bad (for me)
Not JUST you. I've heard all the arguments 100x for "binary logs good/bad", but I feel this was a step down for some puritanical idea that doesn't translate for many to practicality.
after a decade of systemd usage I've never once had to actually think about the format the logs are stored in, or where they're stored; journalctl is actually easier than using tail and cat
The option to fix it is always there, individualized by some rule that is completely different from what a command line interface tends to use, so it's mostly unrecognizable.
It wouldn't really be that bad, except for the fact that it's very hard to be confident it's doing the job you wanted. So it's showing no errors with this service... does that mean the service has no errors?
[1] https://www.digitalocean.com/community/tutorials/how-to-use-...
PartOf= let us still have one top level “lithium” service to start and stop which starts and stops all of the other services, while being able to use journalctl to view them as one log file via a wildcard!
Super nice, it’s been quite lovely.
The thing that got me into it was systemd timers. I really like being able to split up what should run from when should it run. `cron` works best if you desire simplicity above all else, and Slack used `cron` for a very long time before moving to something more custom. But I ultimately don't feel like the learning curve for systemd timers is that bad, and the benefits it can bring (like writing all your echo statements to journalctl instead of some random hard coded logfile in a script) feel very worth it to me.
It may be simple, but it's config file is hard to comprehend compared to a systemd timer. Being able to just use "Friday 17:00" or "hourly" is so much more readable.
systemctl status foo.timer tells you when it has last triggered and when it will triggered next, if it is still enabled.
systemctl status foo.service tells you if it has been triggered from the timer and when, or ran manually with start
I get the conceptual utter simplicity of cron in place of timer + a script in place of service but in practice systemd is much simpler and more consistent to manage.
https://crontab.guru/#*_0/1_*_*_*
Oops. Point taken :)
systemd-analyze calendar "Mon,Tue *-*-01..04 12:00:00"Systemd by Example - https://news.ycombinator.com/item?id=36817510 - July 2023 (11 comments)
Systemd by Example - https://news.ycombinator.com/item?id=30071240 - Jan 2022 (69 comments)
Systemd by Example – The Playground - https://news.ycombinator.com/item?id=30070159 - Jan 2022 (1 comment)
Podmans integration with systemd is much greater, especially when you use quadlets (systemd controlled podman containers).
This was the final straw though: https://forums.debian.net/viewtopic.php?t=154562
I am back to sysvinit and happy.
The UX of systemd tools (systemctl, journalctl) is a bit crap, to be honest, but systemd does what it needs to do well and it is an established component of a modern Linux system.
I'm really tired of people pretending that systemd is the only modern init system around. If you're going to compare it to the status quo, at least don't strawman it. If SysV init scripts are your comparison, it just sounds like you're regurgitating outdated talking points.
Like it or not Systemd breaks the Unix way and for a certain amount of people (I am not pretending to have a measuring stick) the dogma and philosophy are much more important than perceived ease of use or features.
Void does not use SysV init scripts. Neither do the other distros I know of that do not use systemd.
Huh, what do you mean? I'm pretty happy with those.
And this is easily 80% of why I don't like it. I _know_ it's better* but my use-cases have never clashed with the limitations of init.d.
*better: I've recently had an issue that I was trying to troubleshoot while using `journalctl -fu service-name` and it just wasn't giving me enough information/the same content that /var/log/messages was giving me on my other OS (FreeBSD). Perhaps there was a flag I was missing (increase verbosity??) but the UX did get in the way.
Side note: I do chuckle every time I end up using the -fu flag. =)
I also don't care that much about the "Unix philosophy" or whatever, so there's that.
https://seb.jambor.dev/posts/systemd-by-example-part-2-depen...
https://seb.jambor.dev/posts/systemd-by-example-part-3-defin...
https://seb.jambor.dev/posts/systemd-by-example-part-4-insta...
“Write me a service to start this program”
Or
“Write a socket activated service to…..”
Or
“Write a systemd timer that runs every ten minutes”
Etc etc
https://blog.darknedgy.net/technology/2020/05/02/0/index.htm...
>systemd’s developers and almost all public documentation revolve around a “unit-centric” definition of systemd (except when discussing bugs), but my position is that for systemd-the-service-manager the most important construct is not the unit but the job, and hence one ought to understand systemd in a “job-centric” way.
I've got to say, this is probably the best single explainer of the basics of systemd I've read.
I've only recently started fiddling with systemd extensively since I had my Linux-coming-of-age prior to its widespread adoption. I've got to be honest. I've found the experience very confusing and disorientating. The man pages feel really terse, even by manpage standards, and all the information is heavily split up so it's quite difficult to build a mental model of the full system IMO.
Somethint like this would have been magic before, but it was still really helpful to me now with the way it explained things, I feel like it took me most of the way from cargo culting, into more understanding.
Great write-up!
The halt service, along with poweroff, reboot, and kexec get special treatment:
https://www.freedesktop.org/software/systemd/man/latest/syst...
it's my goto way to keep my programming running and have it be restarted if the vm reboots. I use VMs like "pods". I deploy code directly to the VM and run it there along with other programs. I scale up and scale down with: https://www.pulumi.com/
Love Pulumi btw
https://www.reddit.com/r/linux/comments/bsunzw/systemd_vs_in...
Weird that still there is no apress/o'reilly/packt/manning introductory book about systemd.
Packt does have "Linux Service Management Made Easy with systemd" as of Feb 2022, but I've not yet read it.
[0] https://www.pluralsight.com/cloud-guru/courses/mastering-sys...
[1] https://www.packtpub.com/product/linux-service-management-ma...
https://www.freedesktop.org/software/systemd/man/latest/syst...
Well, that's kinda the point of a `man`ual page.
Somewhat complicated scenario, I need to change the mode of a wifi driver (which I do via modprobe), use iw to create a second interface for that wifi device, and finally use networkmanager to create a wifi hotspot based on the host name of the device. We set the hostname based on the mac address of wlan0.
There are a lot of places this can go wrong, and you need a lot of deep knowledge of systemd to get it right.
I start with three services, one to set the hostname, one to create the wlan0_ap device, and one to start up the wifi hotspot using nmcli.
Alright, simple enough, this is the kind of thing systemd is good at, right?
Well no, (and pardon me if my memory isn't perfect on this, it's tricky to debug weird race conditions) sometimes setting the hostname fails. Why? Well sometimes wlan0 doesn't exist, there's a udev rule that renames wlan0 during start up. Apparently I can add the systemd tag to that udev rules, and it will create a `.device` dependency I can Require. Alright, use both require and after keywords just to be sure. Good enough.
Huh, onehsot services are listed as stopped even though they've run. Do some research, find out I need the `RemainAfterExit` keyword yet. Sure, dependencies are working.
Now wlan0_ap device creation sometimes fails. Why? No really, why, this is hard to debug. Well looking around at dmesg it looks like the problem is that networkmanager is trying to do something with the device at the same time I am. Looks like it's enumerating device capabilities at the same time I'm trying to create a new device? Oh well, I'll just add the retry keyword to my service.
Oh, wait, I can't add the retry keyword to one-shot services? Really? Internet tells me to change the service type to simple. Sure, we'll try that. Now that wlan0_ap device exists, let's start our access point.
Also it's still sometimes failing on the hostname step still? The set-hostname command is asynchronous I think? I'll add a line to the start_access_point script that waits until the hostname isn't the default.
Everything is looking good. Oh wait, it's sometimes failing intermittently. What the hell, why doesn't wlan0_ap exist? Service type `simple` considers a service to be started when the command first launches, not when it exit successfully. I've introduced another race condition.
How can I make a service that works like a oneshot, but that you can retry? What, write a shell script and use systemd-notify to say it's complete?
Long story short I ended up just writing a traditional shell-script based init script. Maybe I'm thinking about systemd wrong or something.
Is that something that could be added to modprobe.conf to have it work right without any modprobe calls?
> Huh, onehsot services are listed as stopped even though they've run. Do some research, find out I need the `RemainAfterExit` keyword yet. Sure, dependencies are working.
Yeah I've also been burned not having RemainAfterExit=y in a oneshot unit. What's worse, such an unit can get started multiple times.
> Well looking around at dmesg it looks like the problem is that networkmanager is trying to do something with the device at the same time I am. Looks like it's enumerating device capabilities at the same time I'm trying to create a new device?
To be fair such race condition could have happened in any init system.
> Oh well, I'll just add the retry keyword to my service.
Perhaps having a dependency to start your service before network-manager would have solved this.
> Long story short I ended up just writing a traditional shell-script based init script. Maybe I'm thinking about systemd wrong or something.
It's perfectly fine IMO to have small shell script services for such stuff.
Probably some of the problems are due to network-manager being mostly designed/used for desktop use cases. It also likes to take complete control of all the interfaces so calling iw/ifconfig behind it's back will cause tears (as you found out).
I do agree that race conditions caused by parallel service startup really suck in embedded devices. Systemd really could use a "please be as deterministic as possible" mode for embedded (if it already doesn't have).
Yes, that's how I do it. But that just switches the mode to one that supports virtual interfaces, I still need to add the virtual interface, and I can't do that in a modprobe conf.
>To be fair such race condition could have happened in any init system.
I feel like it's really easy to make that happen under systemd if you're not completely on the ball, where as it's more difficult in things like openrc.
>Yeah I've also been burned not having RemainAfterExit=y in a oneshot unit. What's worse, such an unit can get started multiple times.
You live and you learn, on its own something like that isn't a huge problem.
>Perhaps having a dependency to start your service before network-manager would have solved this.
It was one in a long-chain of similar issues, but yes. There are a lot of ways I probably could have made this work, but at the end of the day I'm still writing shell scripts and not using very many of systemd's "helpful" features.
All the extra complexity is the reason that systemd ships with its own network daemon systemd-networkd to handle that complexity separately. It sounds like you need a .netdev file to create your virtual interface. Also .device files are also handled by networkd. I use them on our embedded devices pretty successfully. Take a look (especially wlan type): https://www.freedesktop.org/software/systemd/man/latest/syst...