undefined | Better HN

0 pointsCJefferson1y ago0 comments

My code isn't dependant on multi-threading at all.

It use fork in Python multiprocess, because many packages can't be "pickled" (the standard way of copying data structures between processes), so instead my code looks like:

* Set up big complicated data-structures.

* Use fork to make a bunch of copies of my running program, and all my datastructures

* Use multiprocessing to make all those python programs talk to each other and share work, thereby using all my CPU cores.

0 comments

nyrikki1y ago

'Threading' is an overload term. And while I didn't know, I was wondering if at the library level, the fact that posix_spawn() pauses the parent, while fork() doesn't, that is what you were leveraging.

The python multiprocessing module has been problematic for a while, as the platform abstractions are leaky and to be honest the POSIX version of spawn() was poorly implemented and mostly copied the limits of Windows.

I am sure that some of the recent deadlocks are due to this pull request as an example that calls out how risky this is.

https://github.com/python/cpython/pull/114279

Personally knowing the pain of fork() in the way you are using it, I have moved on.

But I would strongly encourage you to look into how clone() and the CLONE_VM and CLONE_VFORK options interact, document your use case and file an actionable issue against the multiprocessing module.

Go moved away from fork in 1.9 which may explain the issues with it better than the previous linked python discussion.

But looking at the git blame, all the 'fixes' have been about people trading known problems and focusing on the happy path.

My reply was intended for someone to address that tech debt and move forward with an intentional designed refactoring.

As I just focus on modern Linux, I avoid the internal submodule and just call clone() in a custom module or use python as glue to languages that have better concurrency.

nyrikki1y ago

I found where subprocess moved to posix_spawn() that may help.

https://bugs.python.org/issue35537

My guess is that threads in Cython are an end goal. While setting execution context will get you past this release, fork() has to be removed if the core interpreter is threaded.

The delta between threads and fork/exec has narrowed.

While I don't know if that is even an option for you, I am not seeing any real credible use cases documented to ensure that model is supported.

Note, I fully admit this is my own limits of imagination. I am 100% sure there are valid reasons to use fork() styles.

Someone just needs to document them and convince someone to refactor the module.

But as it is not compatible with threads, has a ton of undefined behavior and security issues, fork() will be removed without credible documented use cases that people can weigh when considering the tradeoffs.

j / k navigate · click thread line to collapse

0 comments

nyrikki1y ago

I am sure that some of the recent deadlocks are due to this pull request as an example that calls out how risky this is.

https://github.com/python/cpython/pull/114279

Personally knowing the pain of fork() in the way you are using it, I have moved on.

But I would strongly encourage you to look into how clone() and the CLONE_VM and CLONE_VFORK options interact, document your use case and file an actionable issue against the multiprocessing module.

Go moved away from fork in 1.9 which may explain the issues with it better than the previous linked python discussion.

But looking at the git blame, all the 'fixes' have been about people trading known problems and focusing on the happy path.

My reply was intended for someone to address that tech debt and move forward with an intentional designed refactoring.

As I just focus on modern Linux, I avoid the internal submodule and just call clone() in a custom module or use python as glue to languages that have better concurrency.

nyrikki1y ago

I found where subprocess moved to posix_spawn() that may help.

https://bugs.python.org/issue35537

My guess is that threads in Cython are an end goal. While setting execution context will get you past this release, fork() has to be removed if the core interpreter is threaded.

The delta between threads and fork/exec has narrowed.

While I don't know if that is even an option for you, I am not seeing any real credible use cases documented to ensure that model is supported.

Note, I fully admit this is my own limits of imagination. I am 100% sure there are valid reasons to use fork() styles.

Someone just needs to document them and convince someone to refactor the module.

j / k navigate · click thread line to collapse