undefined | Better HN

0 pointsCerthas1y ago0 comments

But multiprocessing can't?

I used to write a ton of MPI based parallel python. It's pretty straightforward. But one could easily imagine trying to improve the multiprocessing ergonomics rather than introducing threading. Obviously the people who made the choice to push forward with this are aware of these options, too. Still mildly puzzling to me why threads for Python are needed/reasonable.

0 comments

bdd8f1df777b1y ago

A common pain point for me is parallel data loading and processing for PyTorch or TensorFlow. Multiprocessing has a lot of bugs and pain points to deal with when doing ML. Examples: https://github.com/search?q=repo%3Apytorch%2Fpytorch+multipr.... Most of these issues do not exist in a multithreading world, because resource sharing is trivial in that case.

Since Python leads over any other languages in the ML community, and ML is a hot topic right now, it makes sense for Python developers to secure the lead by making the life of ML developers easier, which is by introducing GIL-less multi-threading.

pdhborges1y ago

It doesn't to be puzzling just read the motivation section of https://peps.python.org/pep-0703/

CerthasOP1y ago

I know that document. It doesn't really answer this point though. It motivates the need for parallelism in slow python by noting that this is important once other performance critical code is in extensions.

But the mai point against multiprocessing seems to be that spawning new processes is slow ...

That single "alternatives" paragraph doesn't answer at all why mp isn't viable for python level parallelism.

I am no longer invested in python heavily, I am sure there are discussions or documents somewhere that go into this more. Might be that it's simply that everyone is used to threads so you should support it for sheer familiarity. All I am saying is it's not obvious to a casual observer.

j / k navigate · click thread line to collapse