Show HN: Simple Log Alerts to Slack (opens in new tab)

This looks very interesting. It seems to work on a regex match, so you'd set what you want to see alerted on and then you receive alerts if that match is made?

I like the opposite method - default alert on everything and develop an allowlist that quietens things down you don't want to hear. This is great for alerting you to unexpected things. And once in a while you actually want to know about some of those things :-)

It may sound very noisy but it's not too bad, especially once you're allowlist is setup. Logcheck[0] is a good tool for this and it runs by default at 2 minutes past each hour, emailing in a report of everything that isn't allowed. I think it matches some regex to what it deems higher threat events and those are always alerted on.

I'll conceed that this method isn't stellar for cattle! And we don't bother with it for things like kubernetes clusters or servers with semi-regular turnover for instance.

For pets and long lived servers that need looking after it's a good tool.

[0]https://logcheck.org/

An interesting approach, especially the hybrid python + perl bundling. Embrace the 'you can't please everyone, all of the time' mantra.

At work we're using a combination of Splunk, Microsoft Sentinel, and Grafana Loki (can really recommend the latter combined with promtail).

There's some curious artefacts in this repo - the systemd service file calling usr/bin/env to a $PYTHON_EXECUTABLE (set in the env file to /usr/bin/python3) but the python script being called starts with '#!/usr/bin/env python' (which on my Debian unstable system launches python2).

Throwing executables under /etc/ is a .. contentious manoeuvre.

And the - so far as I can see - lack of support for handling journald output I suppose is a reflection of this sentiment:

> There’s endless criticism of systemd in the Linux community and I expect it will be properly superseded in the next few years. [...] It’s a curiously un-robust system.

You could probably also use fluentd to do the same thing. It can process the logs locally and sent alerts instead of forwarding logs to a central source.

How to grep the logs for errors and send an email

https://docs.fluentd.org/how-to-guides/splunk-like-grep-and-...

The slack plugin you could use instead of email:

https://github.com/sowawa/fluent-plugin-slack

If most of your stack is written in Python you could also consider setting up a custom log handler (with additional filters or formatters) such as https://pypi.org/project/slacker-log-handler/

The Python logging framework is quite flexible

Hrmm, I'm not sure I'd call Datadog or Splunk "log alerting systems". That's one of n capabilities that involves logs in each of those platforms. If you just want to fire an alert based on a match in a log line, you definitely do not need those platforms, I agree.

I really like this, the simplicity is really refreshing.

Nothing to do with the article, but it would be nice if Slack (or IOS/Android) offered better customization of notifications. Errors and warnings are indistinguishable from regular messages on mobile, which means they often get lost in the shuffle.

Very nice. I think a lot of admins jump straight to the most complicated Elasticsearch/Datadog/NewRelic stack and don't even take a second to build something simpler.

For "unfussy" machines, I like to just set up rsyslog to send high priority syslogs right to email. [1]

Not trying to break your party, but GrayLog2 solves all those problems. Most businesses/startups do not need well marketed DataDog or Logtail.

Free, fast, versatile, adaptable to almost every case

Why use Slack? Slack has rate limits and one will not get the notifications when something real goes down as there will be 100s of messages overloading Slack API.