I gave up on self-hosted Sentry (2024) (opens in new tab)

We self-host sentry in Hetzner, but with a high-end server. 96c, 512gb. It ends up only costing around $300 a month, however with the scale of events that it processes, the managed version would be in the 10's of thousands.

The overhead at low volume is pretty high, but in the higher volumes (25M transactions/24h) it's a massive cost saving for us.

Edit:

There were just some initial headaches with needing to increase kafka partitions and add replications to the transaction processors, otherwise we didn't quite leverage the available compute and the backpressure would fill Redis up until OOM.

FOSS Sentry fork GlitchTip keeps things more simple and self-hosting friendly.

https://glitchtip.com/

This isn't okay - the author is selling their own alternative to Sentry, 'reusing' Sentry's open-source client SDK's, while spreading FUD about self-hosting Sentry.

I've been self-hosting Sentry for over 10 years: Sentry is installed by running `git clone`, editing one or two configuration files, and running ./install.sh. It requires 16GB RAM if you enable the full feature set. That includes automatically recording user sessions on web or mobile for replay, and support for end-to-end tracing (so seeing which database queries your api asks for in response to a button tap in your mobile app).

Sentry has a wonderful self-hosting team that's working hard to make Sentry's commercial feature set available, for free, to everyone that needs a mature error tracking solution. You can talk to them on discord and they listen and help.

also in recent news: https://it-notes.dragas.net/2024/12/28/i-almost-died-for-a-f...

Hey, that's me!

When I posted this myself on Reddit, I said the following:

I've long held off on actually posting this article to a platform like this one (don't bash your competition and all that), but "isn't Sentry self-hosted?" _is_ one of the most asked questions I get, and multiple people have told me this blog-post actually explains the rationale for Bugsink better than the rest of the site, so there you have it.

> we don’t necessarily recommend self-hosting to everyone. In addition to existing hidden costs, as Sentry evolves, our self-hosted version will become more complex, demanding additional types of infrastructure.

Any insights on why Sentry is so complex and needs so much resources? Is collecting, storing, and organizing errors messages and stack traces at scale difficult? Or it's the other features on top of this?

We've just applied a helm chart a while back. It just works. We maybe had like a few incidents over the years, requiring stuff like Kafka queues to be wiped.

The argument that you have to read a sh script doesn't make sense to me. Are you gonna read source code of any software is referenced in this script or any you download too? No? What's the difference between that and a bash script, at the end of the day both can do damage.

Maybe Sentry don't want you to have self-hosted version?

> I’m not going to run a piece of software that requires 16GB of RAM, has a complex installation script, and is known to be a pain to maintain.

This has to be self-hosted eventually either by you or Sentry themselves so the full cost of this is coming down somewhere. The planet is round and all that and there's no getting away from these inefficiencies, but you can kick the proverbial can down the road.

Also, they are incentivized to make the open product as hard to use as possible so color me surprised the self-hosted edition is hard to operate.

The biggest issue I have with these solutions is indeed local debugging.

I use Sentry with most of my clients, and for effective debugging I need to spin my own Sentry in a Docker container which ends up being quite heavy on my machine especially when combined with Grafana and Prometheus.

I'm really unhappy with virtually all monitoring/telemetry/tracking solutions.

It really feels they are all designed to vendor lock you in their expensive cloud solutions and I really don't feel I'm getting my $s back at all. Worst of all those cloud vendors would rather add new features non-stop rather than honing what they currently have.

Their sales are everywhere, I've seen two different clients getting Datadog sales people join private Slacks to evangelize their products.

Both times I escalated to the CTO, both times I ended up suspecting someone in management had something to gain from pushing teams to adopt those solutions.

This absolutely mirrors my experience. Sentry was straightforward to deploy years ago, but now seems like one of the more egregious offenders in the, 'self host-able but increasingly prohibitively complex by design' category.

As others have said, we've [0] found the only practical way to deploy this for our clients is Kuberentes + Helm chart, and that's on bare-metal servers (mostly Hetzner). It runs well if you can throw hardware and engineering time at it, which thankfully we can. But given the option we would love a simpler solution.

[0]: https://lithus.eu

I'm a big fan of self-hosting, but it requires the right infrastructure. You need a robust virtualization solution like Proxmox, equipped with sufficient CPU cores and RAM. For instance, allocating 16GB of RAM shouldn't be an issue if your server has 128GB to share.

In my experience, solutions like Mailcow, which involve multiple services and containers (such as SMTP, IMAP, Redis, SSO, webmail, Rspamd, etc.), work very well. I have extensive experience running these systems, performing backups, restoring data, and updating their ecosystems.

Additionally, I've had a positive experience setting up and running a self-hosted Sentry instance with Docker for a project that spanned several years. However, this experience might be somewhat outdated, as it was a few years ago.

A good APM tool I’ve been using for a few years is Apache Skywalking: https://skywalking.apache.org/

It might not do everything Sentry does but it definitely has helped with tracking down some issues, even production ones and runs in a fairly manageable setup (in comparison to how long even the Sentry self-hosted Docker Compose file is).

What’s more, if you want, you can even use regular PostgreSQL as the backing data store (might not be quite as efficient as ElasticSearch for a metrics use case, but also doesn’t eat your RAM like crazy).

At first glance it's not immediately obvious to me, why would you pick Sentry or Bugsink over something included in the Grafana stack? What's the use use

TL;DR: Not so much "gave up" as "never tried" because I was scared away by 16GB RAM requirement.

FWIW, we've been self-hosting 2 instances (and purchase a third hosted instance), for, it looks like 8 years now, and it's had a few small bumps but hasn't been too bad. Our instances are running with 20GB of RAM, so pretty small. ~90% of updates have gone smoothly, which leaves 10% of updates that have had some sort of problem. Solutions have been found in the issue tracker in all our cases. We are a little afraid of doing updates, but do them every couple of months.

Sentry is an amazing piece of software and it is fantastic that they offer a self-hosing version, and all things considered it is a fairly easy self-host.

I find it very funny that the post is basically complaining about fearmongering by the maker of Sentry to scare people into not hosting themselves.

And then he does the exact same thing, on behalf of Sentry.

I hope he got paid for this. Otherwise it would just be sad.

From the Sentry homepage

> Application monitoring software considered "not bad" by 4 million developers.

Sounds pretty bad to me

This is also my opinion of Backstage. It's positioned as this easy-to-use open source tool for IDPs - but in reality, it's a product that needs a team to maintain it constantly.

> First, there’s a signal being given about the complexity and fragility of the software. A 16GB RAM requirement just screams “this is a big, complex piece of software that will break in mysterious ways”. Not very scientific, I know, but based on experience.

This lines up with my experience self hosting a headless BI service. In "developer mode" it takes maybe 1GB RAM. But as soon as you want to go prod you need multiple servers with 4+ cores and 16GB+ RAM that need a strongly consistent shared file store. Add confusing docs to the mix, mysterious breakages, incomplete logging and admin APIs, and a reliance on community templates for stuff like k8s deployment... it was very painful. I too gave up on self hosted.

As a computer user I do not allow connections to sentry.io

For example, Firefox tries to connect

That's the inevitable evolution of any popular open-source product/company. Light and enterprise-ready is hardly compatible, and you can expect this from any project that has a cloud offering.

I am not sure why there isn't a even lightweight sentry endpoint that does

  - save post body to folders (use uuid as folder name to avoid spam)
  - dir listing, and count number of entries
  - render posted json to html, highlight stacktrace with js
  - download raw json
  - rotate, compress old entries.

I give those requirements to LLM, and I get a pretty much working rust implementation after few tweaks. It uses <5M ram idle.

I still love Sentry, but it’s so enormous now that it isn’t practical to self-host for smaller businesses. A “small” alternative is always great to see!

I’m not sure how I feel about the license though (Polyform Shield, basically use-but-don’t-compete). It’s a totally valid choice – I just wish it would convert to FOSS at some point. (I understand the concern as I’ve had to make a similar decision when releasing https://lunni.dev/. I went with AGPL, but I’m still overthinking it :-)

The amount of shell script that needs to be executed to install is a bit of a no no for me. It also doesn't make sense to spin up a 16GB machine (minimum!) to track the errors on those 4-8GB VPS which are running my production services.

This is what usually happens with self-hosted software that offers a cloud paid alternative. If it's too easy to install and maintain, the cloud version would make no money.

For my self-hosted analytics platform [0], I chose to only use the LAMP stack (PHP/MySQL), as it's one of the easiest to set up and maintain. Yes, it won't be able to track 100 million events per day, but most people don't have that amount of traffic anyway. It runs on the cheapest VPS, and can be installed in many different ways (adding more and more, if the platform is self-hosted only, my goal is to make installation as simple as possible, especially with the free trial, so customers can try it anywhere in a few minutes). Sorry if it sounds like a rant, but it's hard competing with huge corporations promising free "self-hosted options", only to use it as a lead-magnet for the cloud service. I saw this trend growing, even with smaller companies, where their self-hosted one-click app installs fail and they provide no support for it or have no intention to fix the installer.

[0]: https://www.uxwizz.com

I use both hosted and self hosted sentry. I prefer hosted it's less to manage but self hosted isn't too awful as long as you can budget the resources. If you just want the team sentry plan it's going to be cheaper to use hosted. I would only self host if you had to for legal/compliance reasons. In the many years I've been managing sentry we've only had it truly crap out once and that was an upgrade from 9 to their new date versioning basically the whole hosted method changed and it was just easier to start over from scratch.

HyperDX is a great alternative to Sentry, much easier to self-host, and also open source.

It's relatively new and did take some tinkering to make it work properly, so I wrote a short article about it: https://weberdominik.com/blog/self-host-hyperdx

But the feature set and user experience is great!

So many services are going this route. I remembered Reddit had an install script and almost everything worked out of the box 10-15 years ago on a Ubuntu VPS.

I worked in a startup around 2019-2020 where we started using Sentry for distributed tracing. It was a company in the IOT space and we had a lot of events going through. The bill started going in the $3-5000 range just from the tracing, so we decided to host it on our own. When looking at the sheer complexity of the project, I was flabbergasted. We need a name for managing the infrastructure of company based open source, self-hosted solutions. Often times, the right choice would be to choose a different, simpler, open source solution. In this case there are some neat distributed tracing solutions, that are easier to manage.

Software that's only available as x86 binaries and a whole system that needs 16 gigs of memory? It's not surprising that it comes from the corporate world and not the open source world.

I respect the fact that the Sentry people are honest about their teetering tower.

> It’s a ton of docker containers. Things will fail randomly or maybe with a lot of traffic, don’t remember well.

maybe they should put in a system to monitor the docker containers.

I was looking into moving away from managed Sentry for GDPR reasons. I was only using Sentry for notifications about unhandled exceptions in my app. All the fancy APM features I did not use. Grouping of similar exceptions, pretty display of tracebacks were nice but not essential.

In my Django app I wrote a logging handler that stores the log records (including traceback) in a database table. I can inspect the log records through Django admin, and a cron job sends me daily emails saying "X new log records in the last 24 hours" so I know to check them out. And that's it :-)

Of course, this does a lot less than Sentry, and has various limitations (e.g. what if the error is about the database being down...), but it fits my needs.

BTW, IIUC, Sentry in its early beginnings was also doing just that – logging to database: https://github.com/dcramer/django-db-log

It's wild that Sentry themselves are spreading literal FUD about about bad it is to self-host their own product. So confounding that they're so ready to shit on their own product.