jturolla

I'm currently leading a observability team at a 150+ engineer, 150+ services company. We spend the last 6 months building infrastructure around Prometheus, including instrumentation, alerting rules, high availability and scalability/sharding. We have a multi-kubernetes-cluster setup and multi-customer-shards with a high growth rate.

When we started, our observability stack was composed of Riemann, Splunk and thousands of unloved cloudwatch alerts. We felt Riemann was very difficult to work with for our setup and our multiple shards, and we evaluated what options we had, finally choosing Prometheus.

We spend a lot of time pairing with squads so they can learn Prometheus and work on their own creating instrumentations and alerts.

We're in the middle of the rollout for Prometheus. Roughly 50% of the services and squads are using metrics and alerts, and this is on-track to becoming 90% until the end of the next quarter.

Having a metrics and alerting platform ready to use is making engineers much more confident on their services, but we're still facing some incidents that are reported by customer support instead of engineering tools, which could mean either we're missing something, or we just have to double down on the rollout to make sure everything is covered and alerting correctly.

Could you share your thoughts and experience around observability?

Ask HN: Why netflix doesn't have playlists as channels were in the past?

Hi, I run some apps on oceanic.mongohq.com. It's offline right now for over 5 minutes. Anyone experiencing this?

So I just got an email from intercom.io that stated they received a message in my website, but I would only be able to see the message if I renew their service.

What happened here is that I didn't uninstall their gem from my website for the lack of time I have. Therefore, I would expect them to not allow my customers to send them messages anymore, but unfortunately they do, and they are keeping the messages as a hostage.

It's interesting how they market this. They say I have "A message from a potential customer", and then they send me to their subscription page.

I find this type of business approach DIRTY and NASTY. It's terribly immoral to collect data in my website, from my customers, and withhold that as a hostage to get my money.

I hope intercom gets this message. I already ranted for them in a long email. I find this approach illegal, as our business contract is over and they are still collecting data from my website.

I understand many companies have a very aggressive way to get old customers back, but that should never bear the immorality barrier.

Thank you.