undefined | Better HN

0 pointsbandrami2y ago0 comments

OTOH "systemctl start httpd.service" exits with 0 even if apache doesn't start because for some reason "a misconfigured apache didn't start" is considered correct behavior. I don't know that any init has actually solved that problem.

0 comments

10 comments · 4 top-level

throw0101a2y ago· 2 in thread

> OTOH "systemctl start httpd.service"

I've also had issues with My/MariaDB not starting and the output capture by systemd and journald be absolutely useless in debugging the issue (i.e., nothing shows up). Generally resort to running mysqld manually at the CLI to either get some kind of stdout/err output or maybe sent to the error log file.

Of course if there's an error really early in the program/daemon initialization you used to be able to run strace against the start-up script, but that's not possible with systemctl. I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in.

ilyt2y ago

> Of course if there's an error really early in the program/daemon initialization you used to be able to run strace against the start-up script, but that's not possible with systemctl.

I mean it's PITA but you can just add strace command to start of ExecStart...

> I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in.

Yeah we hit that with some units that had maintainers preemptively setting security policy so distro-blessed paths were only writable ones for a given unit.

But on plus side all could be fixed in few lines of overrides vs putting whole init script into the repo

> I've also had issues with My/MariaDB not starting and the output capture by systemd and journald be absolutely useless in debugging the issue (i.e., nothing shows up). Generally resort to running mysqld manually at the CLI to either get some kind of stdout/err output or maybe sent to the error log file.

Do you have systemd unit for that mysql problem ? There is option to redirect stdout/err to logs directly but systemd is too stupid to extract that logs back and show it in the status command.

I definitely saw that issue you mentioned (it almost looked like it wasn't logged anywhere if app exited too quickly). But I don't see it on recent debians at the very least.

    ● mariadb.service - MariaDB 10.5.15 database server
         Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
         Active: active (running) since Mon 2023-05-08 12:41:26 CEST; 1 months 21 days ago
           Docs: man:mariadbd(8)
                 https://mariadb.com/kb/en/library/systemd/
        Process: 2913787 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
        Process: 2913788 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
        Process: 2913790 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemct>
        Process: 2913887 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
        Process: 2913889 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
       Main PID: 2913870 (mariadbd)
         Status: "Taking your SQL requests now..."
          Tasks: 18 (limit: 2340)
         Memory: 303.5M
            CPU: 10h 32min 54.873s
         CGroup: /system.slice/mariadb.service
                 └─2913870 /usr/sbin/mariadbd
    
    May 08 12:41:26 hostname systemd[1]: Starting MariaDB 10.5.15 database server...
    May 08 12:41:26 hostname mariadbd[2913870]: 2023-05-08 12:41:26 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 2913870 ...
    May 08 12:41:26 hostname systemd[1]: Started MariaDB 10.5.15 database server.
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysql' as: /usr/bin/mysql
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: You can use --force if you still want to run mysql_upgrade

throw0101a2y ago

> Do you have systemd unit for that mysql problem ?

Yeah: whatever ships/shipped with Debian. (Which we used at my last job, where I had this issue more than once. Don't run the same stack at my new job.)

ilyt2y ago· 2 in thread

Neither does some init scripts we had to fix so it's 0:0 on this particular issue.

That issue is actually systemd considering success if config and dependencies are okay and service is started; it doesn't wait to return so it doesn't know whether service "works okay now", because it might exit with error 20ms after start.

    Type=notify

fixes that issue but obviously needs support from app themself to signal "I'm up and running now"

Less robust way is to set

    Type=forking

then

* if app exited with okay, it is considered running

* if app exited with not okay, you get error on start

but obviously then app need to be of self-forking type.

It's an iffy problem to solve; I guess systemd could have some wait interval to wait for service that might crash some milliseconds after start but that would delay every start unnecesarily

bandramiOP2y ago

Right, like I said I don't think any init system has actually figured this out but that is what as an admin I actually care about rather than startup times or the aesthetic properties of the configuration files. You would need some kind of semantics that define what it means for a system to be "ready". In practice we tended to lean towards some kind of polling of the service itself (which is at least slightly better than "sleep 3" or whatever).

Now, it's true that if applications would adopt systemd's Notify system this could really help, but this is equivalent to saying "if software were different it might be better".

ilyt2y ago

Yeah its PITA problem.

Like, service can fork, return okay then die after a second coz someone set memory size wrong. No real sensible way for init system to prevent it.

Or takes 5 minutes to load then hits some bad data and crashes.

We just set services on auto-restart, monitor it and flag alert if service is restarting too often.

Arnavion2y ago· 2 in thread

This requires the application to tell the service manager to wait for some signal that it has successfully "finished starting" beyond just "the process was created". There's no standard way to do it. In systemd's case this is done by opting in to inherit a socket that can be used to communicate that information (`Type=notify`, then calling `sd_notify()`).

bandramiOP2y ago

Exactly: like I said below the pain point was never "generate a textual representation of the startup logic appropriate to the init system" but always "concretely define what you mean by a service being started or stopped and how to determine that".

ilyt2y ago

there is one other way

If you set Type=forking

and the app exits with non-zero code on start then that will get propagated back to systemctl start

but notify is far better option even if requiring extra app support.

atomicnumber32y ago

Fwiw nginx does do this correctly. I'm not sure what httpd does differently that it doesn't mimic this behavior.

j / k navigate · click thread line to collapse

0 comments

10 comments · 4 top-level

throw0101a2y ago· 2 in thread

> OTOH "systemctl start httpd.service"

ilyt2y ago

> Of course if there's an error really early in the program/daemon initialization you used to be able to run strace against the start-up script, but that's not possible with systemctl.

I mean it's PITA but you can just add strace command to start of ExecStart...

> I've run into that situation when (e.g.) /var/lib/mysql was a link to a mount point (secondary drive) and SELinux or AppArmour kicked in.

Yeah we hit that with some units that had maintainers preemptively setting security policy so distro-blessed paths were only writable ones for a given unit.

But on plus side all could be fixed in few lines of overrides vs putting whole init script into the repo

Do you have systemd unit for that mysql problem ? There is option to redirect stdout/err to logs directly but systemd is too stupid to extract that logs back and show it in the status command.

I definitely saw that issue you mentioned (it almost looked like it wasn't logged anywhere if app exited too quickly). But I don't see it on recent debians at the very least.

    ● mariadb.service - MariaDB 10.5.15 database server
         Loaded: loaded (/lib/systemd/system/mariadb.service; enabled; vendor preset: enabled)
         Active: active (running) since Mon 2023-05-08 12:41:26 CEST; 1 months 21 days ago
           Docs: man:mariadbd(8)
                 https://mariadb.com/kb/en/library/systemd/
        Process: 2913787 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
        Process: 2913788 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
        Process: 2913790 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemct>
        Process: 2913887 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
        Process: 2913889 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUCCESS)
       Main PID: 2913870 (mariadbd)
         Status: "Taking your SQL requests now..."
          Tasks: 18 (limit: 2340)
         Memory: 303.5M
            CPU: 10h 32min 54.873s
         CGroup: /system.slice/mariadb.service
                 └─2913870 /usr/sbin/mariadbd
    
    May 08 12:41:26 hostname systemd[1]: Starting MariaDB 10.5.15 database server...
    May 08 12:41:26 hostname mariadbd[2913870]: 2023-05-08 12:41:26 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.15-MariaDB-0+deb11u1) starting as process 2913870 ...
    May 08 12:41:26 hostname systemd[1]: Started MariaDB 10.5.15 database server.
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysql' as: /usr/bin/mysql
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: This installation of MariaDB is already upgraded to 10.5.15-MariaDB.
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: There is no need to run mysql_upgrade again for 10.5.15-MariaDB.
    May 08 12:41:26 hostname /etc/mysql/debian-start[2913894]: You can use --force if you still want to run mysql_upgrade

throw0101a2y ago

> Do you have systemd unit for that mysql problem ?

Yeah: whatever ships/shipped with Debian. (Which we used at my last job, where I had this issue more than once. Don't run the same stack at my new job.)

ilyt2y ago· 2 in thread

Neither does some init scripts we had to fix so it's 0:0 on this particular issue.

    Type=notify

fixes that issue but obviously needs support from app themself to signal "I'm up and running now"

Less robust way is to set

    Type=forking

then

* if app exited with okay, it is considered running

* if app exited with not okay, you get error on start

but obviously then app need to be of self-forking type.

It's an iffy problem to solve; I guess systemd could have some wait interval to wait for service that might crash some milliseconds after start but that would delay every start unnecesarily

bandramiOP2y ago

Now, it's true that if applications would adopt systemd's Notify system this could really help, but this is equivalent to saying "if software were different it might be better".

ilyt2y ago

Yeah its PITA problem.

Like, service can fork, return okay then die after a second coz someone set memory size wrong. No real sensible way for init system to prevent it.

Or takes 5 minutes to load then hits some bad data and crashes.

We just set services on auto-restart, monitor it and flag alert if service is restarting too often.

Arnavion2y ago· 2 in thread

bandramiOP2y ago

ilyt2y ago

there is one other way

If you set Type=forking

and the app exits with non-zero code on start then that will get propagated back to systemctl start

but notify is far better option even if requiring extra app support.

atomicnumber32y ago

Fwiw nginx does do this correctly. I'm not sure what httpd does differently that it doesn't mimic this behavior.

j / k navigate · click thread line to collapse