Show HN: Kubetail – Web-based real-time log viewer for Kubernetes (opens in new tab)

I don't know, at that point I think I'd rather use an actual log aggregation system like Loki or Kibana. That way I can search, including on pods that are now gone.

The niche between "easy to get but single-container and no search" on one side and "install with helm but search all containers including historical with full-text and metrics" on the other... seems like a tiny niche to me.

edit: oh you need to install Kubetail cluster-wide too. At least no DaemonSet I guess.

This is really nice. I usually use stern for logs ON a cluster but to aggregate all of the logs we usually use something like fluentd to elasticsearch, which when seeing something like this is super complex for what we usually use it for but it does of course let you search everything.

There is an existing project named kubetail, which is quite popular 3.2K starts https://github.com/johanhaleby/kubetail

Awesome project. I run a Kubernetes cluster on my homelab on 4x Raspberry Pi 4 Bs. Gonna set this up tonight.

I believe there is no persistence, or does it cache in local storage or anything on the client? Would be awesome to have that option for client side storage for perhaps 24 hours.

Demo worked fine until I added the kubetail-demo pod as a source, which crashed my browser tab. Copying the same URL into a new tab loaded the page but stuck in "Loading logs" until it also crashed the tab.

Nice, needs a screenshot in the readme.

Congrats on the launch, nice project! I recently launched https://logdy.dev (OSS: https://github.com/logdyhq/logdy-core) which attempts to address the problem but in a more wide space: any kind of process stdout -> web UI. You can run it with k8s (kubectl logs -f). I'm actually writing a blog post about it as we speak and will definitely mention kubetail as well. Ofc, your project addresses the problem more specifically, I just thought to mention Logdy in case somebody is looking for a swiss-knife solution for all kinds of logs.

How is privacy enforced? Are you planning on maintaining this:

https://github.com/kubetail-org/kubetail

as an open source repo?

Does this work well when set up to use my local machine and my personal credentials?

Wow what a small world. I've been looking to build a tool EXACTLY like this (even after seeing Johan's kubetail project) but kept thinking it was too obvious that somebody must have already built it or kXs ecosystem already has something like this and I was just too noob to find it! Only diff - my fe was going to be htmx :). Kudos.

I suppose it is never too late :)

I've written a [proprietary, though I have permission to open source it] `tailfhttpd`, which is a tiny, trivial HTTP/1.1 server that supports only `HEAD`s and `GET`s of regular files, but with a twist:

  - it supports `ETag`s, with ETags derived from
    a file's st_dev, st_ino, and inode generation
    number
  - it supports setting *some* response headers via
    xattrs (e.g., ETag, Content-Type, Vary,
    Cache-Control, etc.)
  - it supports conditional requests (i.e.,
    `If-Match:`, `If-None-Match:`,
    `If-Modified-Since:`)
  - it supports `Range:` requests
  - for `Range: bytes=${offset}-` `GET`s
    the response does not finish (i.e.,
    final chunk is not sent) until one of
     - the file is unlinked
     - the file is renamed
     - the server is terminated
    using inotify to find out about file
    unlinks/renames

It does this using `epoll`, `inotify`, and `sendfile()`, with multiple fully-evented, async-I/O-using processes, each process being single-threaded. It is written in C in continuation passing style (CPS) with tiny continuations, so its memory footprint per client is also tiny. As a result it is blazingly fast, though it needs to be fronted with a reverse proxy for HTTPS (e.g., Nginx, Envoy, ...), sadly, but maybe I could teach it to use kssl.

I use it for tailing logs remotely, naturally, and as a poor man's Kafka. Between regular file byte offsets, ETags, and conditional requests one can build a reliable event publication system with this `tailfhttpd`. For example, and event stream can name the next instance ({local-part, ETag}) then be renamed out of the way to end in-progress `GET`s, and clients can resume from the new file.

With a few changes it could "tail" (watch) directories, and even allow `POST`ing events (which could be done by writing to a pipe the reader of which routes events to files that get served by `tailfhttpd`).

Because `tailfhttpd` just serves files, and because of the ETag thing, conditional requests, and xattrs, it's very easy to build more complex systems on top of it -- even shell scripts will suffice.

This chunked-encoding, "hanging-GET" thing is so unreasonably effective and cheap that I'm surprised how few systems support it.

I've visions of rewriting it in Rust and supporting H2 and especially H3/QUIC to reduce the per-client load (think of TCP TCBs and buffers) even more, and using io_uring instead of epoll for even better performance.

Oh, and this approach is fully standards-compliant. It's just a chunked-encoding, indefinite-end ("hanging") GET with all the relevant (but optional) behaviors (ETags, conditional requests, range requests, even the right end of the byte-range being left unspecified is within spec!).

Nice job