A different and often better way to downsample your Prometheus metrics (opens in new tab)

(NB: post author here)

Great question. We support average of averages by storing the intermediate state of the aggregate (for average that's the sum and count) so we could cleanly re-aggregate.

Eventually, we'll be able to incrementally update the aggregate if we backfill even if the raw data is no longer available. That's not implemented yet though, so backfill only updates the aggregate if the raw data is still around by re-computing the intermediate state of the aggregate off the raw data for affected buckets. For most cases that isn't actually an issue since most people have a longer data retention period than backfill horizon.

You avoid average-of-averages by storing multiple summaries. For example, you don't compute and store average, you compute and store sum and count.

Indefinite persistence of time-series data "as-is", is a somewhat different use-case from putting them in a data warehouse so you can efficiently do rollups to them. Timescale seems to be useful for the latter, but I'm not sure it offers too much value for the former.

I believe the state-of-the-art for plain-old Prometheus data retention is https://thanos.io/ — my understanding is that it's a Prometheus remote storage integration (https://prometheus.io/docs/prometheus/latest/storage/#remote...) that archives time-series data from Prometheus into an object-store, and then fetches/streams chunks back out from said object-store to serve requests.

You could use it locally on your NAS, by running a Minio instance on there. But IMHO there wouldn't be much point in doing that, over just keeping all the data in Prometheus's own internal storage.

RRD is a terrible format. It's not compact, quickly becomes a burden when you have lots of cardinality, there's no metadata, HA is a joke and visualisation tools are basically non-existent. You basically need a whole set of extra tooling for metadata, visualisation, HA, querying to come even close to anything usable.

If RRD wasn't so terrible there wouldn't have been a myriad of replacements.

Interesting - it looks like this is the only past HN thread about it:

Beyond NoSQL: Using RRD to store temporal data - https://news.ycombinator.com/item?id=2742486 - July 2011 (18 comments)

I found a few other tiny threads asking about replacements, and that was it.

Prometheus without any supporting tooling isn't really designed for long term storage as I understand it, however it is built to support long term storage and querying via its remote read/write protocol. Prometheus will write data to remote storage, and can delegate queries to that storage, rather than using its own local storage as it does by default.

Of the various tools that expose the remote read/write APIs, I like the looks of Promscale/TimescaleDB the most so far, but other options like Thanos might make more sense if you need to collect metrics from a bunch of Prometheuses. That said, maybe you can still use Promscale/TimescaleDB with Thanos as the storage backend, I can't recall the details on its requirements though, so it might not be suitable for that case. For my own use cases though, Promscale is a great solution.

(Timescale co-founder)

Thank you for noticing :-)

This is really a testament to all of the amazing products, new features, R&D, and overall work that the team has been shipping.

We are firing on all cylinders. Move fast without breaking things :-)

If this looks like fun to anyone - we're hiring!

Come and help us build the next great database company:

https://www.timescale.com/careers