undefined | Better HN

0 pointsaqme283y ago0 comments

When we used Sidekiq in production, not only did I never see crashes that lost us jobs, but there are also ways to protect yourself from that. I highly recommend writing your jobs to be idempotent.

0 comments

phamilton3y ago

Idempotence doesn't solve this problem. The jobs are all idempotent. The problem is that jobs will never be retried if a crash occurs.

This doesn't happen at a high rate, but it happens more than zero times per week for us. We pay for Sidekiq Pro and have superfetch enabled so we are protected. If we didn't do so we'd need to create some additional infra to detect jobs that were never properly run and re-run them.

ransackdev3y ago

Or install an opensource gem[1] that recreates the functionality using the same redis rpoplpush[2] command

[1] https://gitlab.com/gitlab-org/ruby/gems/sidekiq-reliable-fet...

[2] https://redis.io/commands/rpoplpush/#pattern-reliable-queue

aqme28OP3y ago

Fair enough about idempotence.

I'm still confused about what you're saying though. You're saying that the language of "enhanced reliability" doesn't reflect losing 2 jobs over about 50*7 million (from your other comment)?

And that if you didn't pay for the service, you'd have to add some checks to make up for this?

That all seems incredibly reasonable to me.

danenania3y ago

Crashes are under your control though. They’re not caused by sidekiq. And you could always add your own crash recovery logic, as you say. To me that makes it a reasonable candidate for a pro feature.

It’s hard to get this right though. No matter where the line gets drawn, free users will complain that they don’t get everything for free.

Mavvie3y ago

How are crashes under your control? Again they aren't talking about uncaught exceptions, but crashes. So maybe the server gets unplugged, the network disconnects, etc.

danenania3y ago

To me 'crash' means any unexpected termination, whether it's caused by an uncaught exception, OOM, or hardware/network issues.

I guess you can say that hardware issues on your host aren't under your control, but it's under your control to find a host that doesn't have these issues. And not even a full-on ACID database is going to be 100% reliable if you yank the power cord at the wrong moment.

1 more reply

pselbert3y ago

Jobs may crash due to VM issues or OOM problems. The more common cause of "orphans" is when the VM restarts and jobs can't finish during the shutdown period.

j / k navigate · click thread line to collapse

0 comments

phamilton3y ago

Idempotence doesn't solve this problem. The jobs are all idempotent. The problem is that jobs will never be retried if a crash occurs.

ransackdev3y ago

Or install an opensource gem[1] that recreates the functionality using the same redis rpoplpush[2] command

[1] https://gitlab.com/gitlab-org/ruby/gems/sidekiq-reliable-fet...

[2] https://redis.io/commands/rpoplpush/#pattern-reliable-queue

aqme28OP3y ago

Fair enough about idempotence.

I'm still confused about what you're saying though. You're saying that the language of "enhanced reliability" doesn't reflect losing 2 jobs over about 50*7 million (from your other comment)?

And that if you didn't pay for the service, you'd have to add some checks to make up for this?

That all seems incredibly reasonable to me.

danenania3y ago

It’s hard to get this right though. No matter where the line gets drawn, free users will complain that they don’t get everything for free.

Mavvie3y ago

How are crashes under your control? Again they aren't talking about uncaught exceptions, but crashes. So maybe the server gets unplugged, the network disconnects, etc.

danenania3y ago

To me 'crash' means any unexpected termination, whether it's caused by an uncaught exception, OOM, or hardware/network issues.

1 more reply

pselbert3y ago

Jobs may crash due to VM issues or OOM problems. The more common cause of "orphans" is when the VM restarts and jobs can't finish during the shutdown period.

j / k navigate · click thread line to collapse