Using GNU parallel (YouTube tutorials) (opens in new tab)

(youtube.com)

2 pointsuranium2356y ago2 comments

2 comments

2 comments · 1 top-level

stephenr6y ago· 1 in thread

We've briefly used parallel for an internal tool when the site went "multisite" (i.e. multiple parallel instances that sometimes need to all do the same thing, such as post-deployment tasks).

We currently have a patch to remove it, and instead use regular task backgrounding and the `wait` shell builtin.

The way output is buffered was not particularly conducive to what we wanted to achieve.

MachineMartin6y ago

THE way? It sounds as if GNU Parallel only buffers in one way.

GNU Parallel can handle output in a few ways:

  * --ungroup: With -u output GNU Parallel does not touch the output and output may mix
  * --line-buffer: This buffers a single line from each process.
  * --group: This buffers in files in $TMPDIR. $TMPDIR can point a RAM disk, if you want to buffer in RAM.
  * --sqlworker: This buffers in memory before saving output to an SQL server.

If you are using '&' + 'wait', it pretty much has the same effect as `--ungroup`.

j / k navigate · click thread line to collapse