undefined | Better HN

0 pointsAnimats6y ago0 comments

And thus Borg was born.

Yes. Basic Dijkstra producer-consumer one-way queues are simple and elegant. But the error cases are hell.

OK, so if the consumer wants to make the producer stop, it sets an abort flag. That's a shared variable. (Does it need a lock? Does it need a lock or fencing on ARM, which has weaker memory concurrency guarantees than x86?)

So, producer sees the abort flag set and closes its end of the channel, right? Maybe not. Maybe the channel is full, and the producer is blocked on a send. OK, so, when the consumer aborts, it has to enter a drain loop, doing receives and discarding the result until it gets an EOF/channel close. That will unblock the producer, and then the consumer can exit.

But what if the producer doesn't have anything to send right now? Now the consumer is stuck waiting for the producer to send more data, and the consumer can't exit.

Should the consumer close the producer's end of the channel after setting the abort flag? Now there's a race condition between checking the abort flag and sending. Add a critical section that covers both actions? Now the producer can be blocked while in a critical section. This creates a potential deadlock if the consumer wants to set the abort flag while the producer is inside the critical section. Now both ends are blocked and there's a deadlock.

But it all looked so simple in theory!

0 comments

4 comments · 1 top-level

codr76y ago· 3 in thread

If you insist on making it as difficult as possible, I guess.

Otherwise the producer decides when the show stops, by simply closing the channel.

fovc6y ago

The ECMAScript TC came to the same conclusion with promises and fetch. Instead of adding an `.abort` method to promises for the consumer to use, you need to wire in an AbortController so that the producer (fetch) can close the channel. The consumer then sees an AbortError. It's a bit more work to wire up than simply calling `p.abort()`, but it saves you from having to reason through some hairy race conditions.

earthboundkid6y ago

Yes, exactly. The workers can send results to the manager, but the manager does all the management. Anything else gets too hairy, too fast to reason about. I wrote a blog post about it: https://blog.carlmjohnson.net/post/share-memory-by-communica...

AnimatsOP6y ago

Does that scale to a pipeline, a chain of producers and consumers?

What happened with Go, I suspect, is that the syntax came first. Go uses a "<-" operator to talk to channels. There's no place for an error status with that operator. Hence the problem.

UNIX-type pipelines have SIGPIPE to unwind things when something downstream aborts. Abort has to propagate back up the chain. I tend to argue that the workarounds for not having exceptions are worse than the problems of exceptions.

j / k navigate · click thread line to collapse