In a proper programming language, we'd have something like
parallel [1..5], i => { sleep random()*10+5; possibly_flaky i }
// [{"Seq": 4, "Host": ":", "Starttime": 1692491267...
Instead, the shell environment forces programs to invent their own parameter separator (:::), a templating format ({1}), and a way to output a list of structures (CSV-like). You can see the same issues in `find`, where the exec separator is `\;`, the template is `{}`, and the output is delimited by \n or \0. And `xargs` does it in yet another different way.
It's very hard to acquire and retain mastery over a toolbox where every tool reinvents the basics. If you ever found yourself searching "find exec syntax" multiple times in a week, it's not your fault.
As for alternatives, I'm a fan of YSH[1] (Javascript-like), Nushell[2] (reinvented from first-principles for simplicity and safety) and Fish[3] (bash-like but without the footguns). Nushell is probably my favorite from the bunch, here's a parallel example:
ls | where type == dir | par-each { |it|
{ name: $it.name, len: (ls $it.name | length) }
}
It isn't even just the newer shells that have solved this, zsh also has a solution out of the box¹. The extensive globbing support in zsh can largely replace `find`, and things like zargs allow you to reuse your common knowledge throughout the shell.
For example, performing your first example with zargs would use regular option separators(`--`), regular expansion(`{1..5}`), and standard shell constructs for the commands to execute.
I'll contrive up an example based around your file counter, but slightly different to show some other functionality.
f() { fs=($1/*(.)); jo $1=$#fs }
zargs -P 32 -n1 -- **/*(/) -- f
Edit to add: I'm not suggesting zargs is a replacement for parallel, but if you're only using a small subset of its functionality then it may be able to replace that.
¹ https://zsh.sourceforge.io/Doc/Release/User-Contributions.ht...
Spending extra time doing simple things — because you need to Google e.g. "how to pass multiple space-separated arguments from a string to a command" — is also a waste of time.
Honest question, as I’m struggling to leave the shell environment once the program gets too large. I could use Perl, but $? and the likes get quickly out of hand. Python’s support for pipes was difficult last time I used it, but that may have changed. What would you recommend?
GNU xargs implements limited parallelization, and is compiled C. This functionality is present within busybox, including the Windows version.
https://www.linuxjournal.com/content/parallel-shells-xargs-u...
GNU Parallel will have much greater functionality, but it will not reach as far as xargs.
If you feel like the answer is rewriting the shell, the answer is practically never rewriting the shell. It's learning to use it.
parallel 'sleep {= $_=rand()*10+5; =} ; possibly_flaky {}' ::: {1..5}
Contrast that with GPU shaders where one C-style loop operates on buffers separate from system memory, and can't access system services like network sockets or files. GPUs have around 32 or 64 physical cores, so theoretically that many shaders could run simultaneously, although we rarely see that in practice. And we'd need bare-metal drivers to access the GPU cores directly, does anyone know of any?
The closest thing now is Apple's M1 line, but it has specialized NN and GPU cores, so missed out on the potential of true symmetric multiprocessing.
The reason I care about this so much is that with this amount of computing power, kids could run genetic algorithms and other "embarrassingly parallel" code that solves problems about as well as NNs in many cases. Instead we're going to end up with yet another billion dollar bubble that locks us into whatever AI status quo that the tech industry manages to come up with. And everyone seems to love it. It reminds me of the scene in Star Wars III when Padme notes how liberty dies with thunderous applause.
2) Shared resources (in-memory mutable data, hardware devices) mean the ratio of contention to CPU work goes up when you have more cores.
3) Cores on a single die need to share the same constraints - thermal limits and transistor count. So you're best off having enough powerful cores to get you to a sweet spot of single-core performance vs multi-core parallelism.
4) It's hard to provide a performant and useful many-core machine model. Cache coherence makes it easier to program a many-core machine, but limits performance. Without it, you're stuck with distributed systems-style problems.
An AMD GPU is a grid of independent compute units on a memory hierarchy. At the fine grain, it's a scalar integer unit (branches, arithmetic) and a predicated vector unit, with an instruction pointer. Ballpark of 80 of those can be on a given compute unit at the same time, executed in some order and partially simultaneously by the scheduler. GPU has order of 100 compute units, so that's ~8k completely independent programs running at the same time.
You've got a variety of programming languages available to work with that. There's a shared address space with other GPUs and the system processors, direct access to system and GPU local memory. Also some other memory you can use for fast coordination between small numbers of programs.
There's a bit of a disconnect between graphics shaders, the ROCm compute stack and what you can build on the hardware if so inclined. The future you want is here today, it just has a different name to what you expected.
If there's no straightforward way to do that, then I'm afraid that hardware represents a huge investment in the wrong direction.
Because a GPU can be built from the general-purpose multicore CPU I'm talking about. But a CPU can't be built from a GPU.
What I'm getting at is that if I have to "drop down" to an orthodox way of solving problems, rather than being able to solve them in the freeform way that my instincts leads me, then I will always be stifled.
Intel ca. 2010, probably
The real problem we are facing is that our programming models aren't parallel by default.
>By Moore's Law, we could have had MIPS machines with 1000 cores around 2010, and 100,000 to 1 million cores today, for under $1000.
You can have 10000 RISC-V cores on an FPGA but nobody cares. Why? Because even a bit serial processor (that means it processes one bit per clock cycle, or 32 clock cycles for a 32 bit addition) runs into memory bandwidth limitations very quickly if you have enough of them. Main memory is very slow compared to registers and caches. The only way to utilize this many cores is by having a workload that is entirely latency bound. Your memory access pattern is perfectly unpredictable. The moment you add caching, the number of cores you can have shrinks dramatically and companies like AMD are not slimming down their CPUs, they are adding more and more cache. Their highest end processors have almost a gigabyte of cache.
I agree about the programming models not being parallel by default, and that's one of the things that I specifically rail against in most of my comments. MATLAB/Octave is a good introduction to what parallel programming could be. Also the endless doubling down on large caches, because the multicore design I have in mind would mostly eliminate cache and use that die area for cores and local memories.
I think we're slightly talking past each other here though. The CPU I want to build would have around 10-256 cores on 90s tech. So the same transistors holding 1 Pentium Pro would allow for 1-2 orders of magnitude more MIPS or RISC-V cores and local memories. The design is so simple that I think that's why it was missed by the big fabs.
Today there's little demand for 1000+ cores, but that's partly because nobody can see what they could do. But we can't design the thing, because the status quo has us all working pedal to the metal in first gear to make rent. It's a chicken and egg problem that has a lower likelihood of being solved as time goes on. Which is why I think we're on the wrong timeline, because if the system worked then actual innovation would become more accessible over time.
At least, 6000+ 32-bit multiplies per clock tick on ~2GHz+ clocks. Even cheap GPUs easily are 2000+ shaders.
> GPUs have around 32 or 64 physical cores
NVidia SMs and AMD WGPs are not "cores", they are... weird things. They have many shaders inside of them and have huge amounts of parallelism.
As far as grunt-work goes, a "multiplier unit" (literally A x B) is perhaps the most accurate count to compare CPU cores vs GPU "cores", because the concept of CPU-core vs GPU WGP / SM is too weird and different to directly compare.
Split up that WGP / SM into individual multipliers... and also split up the ~3 64-bit multipliers or ~48 CPU SIMD multipliers per core (3x 512-bit on Intel AVX512 cores), and its perhaps a more fair comparison point.
---------
Back 20 years ago, you'd only have 1x multiplier on a CPU core like a Pentium 4, maybe as many as 4x with the 128-bit SSE instructions.
But today, even 1x core from Intel (3x 512-bit SIMD) or 1x core from AMD (4x 256-bit SIMD) has many, many, many more parallel elements compared to a 2004-era CPU core.
They aren't weird things. They are the equivalent of CPU cores. By your logic CPU cores aren't CPU cores, "they are... weird things" because of SMT.
https://en.wikipedia.org/wiki/Cray_X-MP
Price US$7.9 million in 1977 (equivalent to $38.2 million in 2022)
Weight 5.5 tons (Cray-1A)
Power 115 kW @ 208 V 400 Hz[1]
CPU 64-bit processor @ 80 MHz[1]
Memory 8.39 Megabytes (up to 1 048 576 words)[1]
Storage 303 Megabytes (DD19 Unit)[1]
FLOPS 160 MFLOPS
Many thanks to Ole Tange for developing the wonderful tool and helping the users on Stack Overflow sites to this day.
Shameless plug, I am developing a tutorial on GNU Parallel to be presented at eScience conference in Cyprus this year: https://www.escience-conference.org/2023/tutorials/gnu_paral...
mkdir /tmp/g
seq 1 10 | tr \\n \\0 |
xargs -0n2 -P4 bash -c 't=$EPOCHREALTIME; sleep $((RANDOM%5)); echo "$@" >/tmp/g/$t' d0
cat /tmp/g/*
xargs -P "$(nproc)" --process-slot-var=s sh -c 'grep X "$@" >>/tmp/g.$s' d0
cat /tmp/g.*
It’s like xargs with sane defaults and a couple tricks of its own.
It looks like the actual name of the task-spooler command on Debian after install is “tsp”, not “ts”. So no collision :)
Now it just remains to be seen if the package by default allows the tasks to continue to run after I log out, or if systemd will annoyingly kill the tasks after I disconnect from ssh the same way systemd annoyingly kills my “screen” sessions when I disconnect ssh, and there is some cumbersome thing you have to do on each of your systems to have systemd not kill “screen” :(
And still answering every xargs Stackoverflow question with "you should use GNU Parallel" instead of answering the question? That really gets old quickly when googling for xarg answers.
These are just some of the reasons I'll never use parallel. xargs is perfectly fine for most usecases, and it can do everything I need it to.
IIRC the citation notice was cleared by Stallman as GPL compatible. I’d be surprised if anyone’s paid, I assumed that’s rhetoric to imply the value of a citation, or lack of citation, for anyone publishing scientific works.
> These are just some of the reasons I’ll never use parallel.
Hey I’ve actually ranted on HN before about the citation notice (e.g. https://news.ycombinator.com/item?id=15319715) - in part because I find the language of the notice a little misleading; it’s not tradition to write citations for tools used to conduct research, and it’s a requirement (not just tradition) to cite academic sources. If I used parallel to speed up some calculations, that doesn’t justify an academic citation. I don’t cite bash or python or C++ when I write papers either. On the other hand, if I’m writing a computer science paper about how to parallelize code, and especially if I compare it to GNU Parallel, then a citation isn’t optional, and I don’t need a guilt trip to add one, it’ll get requested in review, and rejected without. Is there even a journal publication to cite? (Edit: found it - the request is to cite an article in USENIX magazine.) So I find the notice a little irritating and I’m not sure who it’s aimed at exactly, or what the history of Ole feeling snubbed by scientists really is. Maybe some people were trying to compete with GNU Parallel and failing to cite it? Maybe Ole is paid by an organization that appreciates citations and will continue to fund development on Parallel if there’s evidence of it’s use in academia?
That said, GNU Parallel really is totally awesome, the documentation is amazing, and the citation notice is a one-time thing you can silence permanently. I don’t think the notice is a good reason to never use Parallel, and I do think Parallel is worth using, FWIW.
Thia is true, but it also makes it very hard for academics and PhD students who mainly write software over papers. They get no citations and eventually have to leave academia.
If we had a better practice of citing central software we use - at least the academic software that wants to be cited - we could have a more flourishing ecosystem of such software funded by the universities.
Academics seem to have a very blinkered attitude to this. I wrote some software that was popular for a while in a niche field, and people were forever asking me to waste my time by 'publishing' the manual in some pointless journal so that they could cite something and give me credit. Writing useful software counts for less in that world than publishing another pointless paper that no-one will read.
This doesn't scale. Imagine if all the software you used nagged you and had their own individual methods to silence them. I don't think this would be reasonable.
What makes this particular software so special?
Do you have a source for this? Im confused by this, as the GPL section 7 is pretty clear that additional restrictions are effectively void. I suppose it’s technically not contrary to the GPL to idly state those restrictions, but it is contrary to the GPL to expect them to do anything. If the author is deliberately including an impotent clause in the hope that people will follow it anyways, I feel that trying to confuse or scare people into doing something the GPL gives them explicit permission to do is contrary to the spirit of the GPL.
Furthermore, trying to retaliate against people who (as permitted by the GPL) remove the citation notice, as the author here has done, seems very contrary to the spirit of the GPL.
I really hope that whomever adjudicate these disputes regarding licence agreements doesn't care what a random person says about it.
https://gitlab.archlinux.org/archlinux/packaging/packages/pa...
Debian too (thanks to iib for pointing this out)
https://salsa.debian.org/med-team/parallel/-/tree/master/deb...
And looks like the author is aware of both:
https://gitlab.archlinux.org/archlinux/packaging/packages/pa...
But yeah, if guy wants to have the name of the app mentioned there is a BSD license for that I thin...
- # *YOU* will be harming free software by removing the notice. You
- # accept to be added to a public hall of shame by removing the
- # line. That includes you, George and Andreas.
[david@pc ~]$ echo foo | parallel echo
Academic tradition requires you to cite works you base your article on.
If you use programs that use GNU Parallel to process data for an article in a
scientific publication, please cite:
Tange, O. (2023, July 22). GNU Parallel 20230722 ('Приго́жин').
Zenodo. https://doi.org/10.5281/zenodo.8175685
This helps funding further development; AND IT WON'T COST YOU A CENT.
If you pay 10000 EUR you should feel free to use GNU Parallel without citing.
More about funding GNU Parallel and the citation notice:
https://www.gnu.org/software/parallel/parallel_design.html#citation-notice
To silence this citation notice: run 'parallel --citation' once.
foo
[david@pc ~]$ > Tange, O. (2023, July 22). GNU Parallel 20230722 ('Приго́жин').
https://en.wikipedia.org/wiki/Yevgeny_Prigozhin
edit: all releases are named after current political events:
You can add any message you want into your GPL program. Also, a GPL program does not have to be free.
This has nothing to do with the GPL. You can say in your program that 'by using this software you agree that you're a a cat' and license it under the GPL.
That does not mean the GPL relates to cats in any way.
> All other non-permissive additional terms are considered “further restrictions” within the meaning of section 10. If the Program as you received it, or any part of it, contains a notice stating that it is governed by this License along with a term that is a further restriction, you may remove that term.
Debian explains this further in their patch file.
If I put the GPL in my software and add a file next to it that says "Also you can't use this software if you make more than $100k/year", I've pretty clearly added an additional clause that's incompatible with the GPL.
It says "please cite" and "feel free to not cite if you pay".
It doesn't say "must cite" or "you may only not cite if you pay".
IANAL, but it doesn't seem like it would interact with the GPL at all. So the worst that could be said is that the implementation is annoying or in poor taste.
I agree that in this case it’s likely not enforceable/binding especially since the GPL specifically allows you to ignore those terms. Hopefully that’s legally binding in your jurisdiction vs the other party.
But it’s a straightforward clickwrap agreement, even if the terms are non-monetary the GPL simply doesn’t allow these at all. Can’t place any stipulations on how the user uses the software.
What gives?
The startups you mention actually changed their license. That's what GNU Parallel would have to do to make this extra condition ok, but he won't do it because being a GPL-licensed GNU tool is critical to its popularity in the first place.
Part of the issue is that Ole’s citation notice doesn’t appear at first glance to some people to be compatible with the GPL. You have to read the language carefully, and read the history of GNU Parallel’s citation notice, to understand that the notice is not a licensing term.
Another part of the issue is that the notice doesn’t sound like someone just trying to make a living. It sounds like a demand or even a veiled threat, and one that is inflicted on everyone, not just academics. It’s not exactly clear about what the legal requirements even are.
I’m in favor of Ole getting citations, and I’m in favor of his right to ask. But the way it’s being asked for rubs me the wrong way a little bit, and it’s rubbed other people the wrong way a little bit ever since it was introduced. BTW, the whole reason it seems like all hell breaks loose, and the only reason this matters is precisely because the software is widely used. If it wasn’t widely used and it didn’t sit under the GNU umbrella, you’d never hear about this.
[0] https://gitlab.archlinux.org/archlinux/packaging/packages/pa...
https://lists.gnu.org/archive/html/parallel/2013-11/msg00006...
https://GitHub.com/shenwei356/rush
As you mention xargs has parallel capabilities and gargs is Apache licensed software that fixes some of xargs shortcomings:
https://GitHub.com/brentp/gargs
No reason to use gnu parallel.
Where you get the "or pay 10000€" part from? As far as I remember, the software, unless told otherwise, asks authors of scientific papers to cite GNU parallels if they used it when writing their papers. And it doesn't force it, it's not part of the license, but asks you to do so as it's academic tradition to use citations.
You could just ignore the citation and not break the license, no one would think less of you for doing so.
Most likely from the manpage:
If you use --will-cite in scripts to be run by others you are
making it harder for others to see the citation notice. The
development of GNU parallel is indirectly financed through
citations, so if your users do not know they should cite then you
are making it harder to finance development. However, if you pay
10000 EUR, you have done your part to finance future development
and should feel free to use --will-cite in scripts.
If you do not want to help financing future development by letting
other users see the citation notice or by paying, then please
consider using another tool instead of GNU parallel. You can find
some of the alternatives in man parallel_alternatives.
"If you pay 10000 EUR you should feel free to use GNU Parallel without citing."
find ./ -type f -iname '*.jpg' -size +1M -print0 | parallel -0 mogrify -format webp -quality 80 {} xargs -n 1 -P 8 #!/usr/bin/env bash
set -e
main() {
if [ "$1" = "handle-file" ]; then
shift
handle-file "$@"
else
find . \
-type f \
-not -path '*/optimized/*' \
-print0 \
| xargs \
-0 \
-L 1 \
-P 8 \
-I {} \
bash -c "cd \"$PWD\" && \"$0\" handle-file \"{}\""
fi
}
handle-file() {
echo "handle-file $1 ..."
}
main "$@"* Grouped output (prevents one process from writing output in the middle of another's output) * In-order output (task a output first, task b output second even though they ran in parallel) * Better handling of special characters * Remote execution
More here: https://www.gnu.org/software/parallel/parallel_alternatives....
I was a little put off by the annoying/scary citation issue mentioned by another commenter, so I am not sure I will use parallel.
I want to pipe the output of parallel processes into a utility that I wrote for progress printing (https://github.com/titzer/progress), but I think that neither of these solutions work; my progress utility will have to do this on its own.
Edited to add: finally got signed in to work, you create the script via:
parallel --embed > scriptname.sh
In this case, we don't have control over the docker images used to build our apps.
See also https://hn.algolia.com/?q=gnu+parallel for other related discussions.
For example, translating a large list of IPv4 ranges into a standard format for a firewall rule-set parser:
cat ~/blacklist.p2p | parallel --ungroup --eta --jobs 20 "ipcalc {} | sed '2!d' " | grep -Ev '^(0.|255.|127.)' >> ~/blacklist_p2p_converted
Makes an annoyingly slow task tolerable, as parallel doesn't block while fetching to preserve order. We probably should rewrite this to be more efficient, but this task is run infrequently.
Happy computing =)
While there are cases when it makes sense to stick to what is specified by POSIX, there are also cases when the POSIX specification is so obsolete that using POSIX instead of some free ubiquitous programs is a big mistake.
Among these latter cases are writing scripts for a POSIX shell instead of writing them for bash and using xargs instead of parallel.
Second paragraph: I want to test my test-tester.
OP 100% fell down a rabbit-hole.
"they execute extensive scenarios against a live service over HTTP"
Any time I've seen people think they've needed to test live services, over HTTP... it means that there are far deeper issues.
Curious if anyone else has experiences with it, honestly been surprised at how little I've heard about it
I was expecting something simple as 'parallel -j10 curl https://whatever' but couldnt find the right syntax in less time that took me to prepare a dirty shell script that did the same.
wrk -t2 -c100 -d30s -R2000 http://127.0.0.1:8080/index.html
Some distros include `ab`[2] which is also good, but wrk2 improves on it (and on wrk version 1) in multiple ways, so that's what I use myself.
parallel -j 10 curl 2> /dev/null \
::: $(for i in {1..10};do echo 'https://whatever.com';done) /usr/bin/parallelBut because of the mini learning curve on each use and because I find I need a little more boiler plate to use parallel, I use xargs -P more often, only using parallel when I need its special features (e.g. multiple hosts or collating the output streams).
Oh also, parallel itself can be a bit of a resource hog. (Obviously that depends a lot on how you're using it-- but I mean in cases where xargs' usage is unnoticeable I sometimes have to change the size of my jobs to get parallel out of the way).
Thanks for the link to the book: https://zenodo.org/record/1146014
Here are a bunch of examples: https://www.gnu.org/software/parallel/parallel_examples.html
A fun one I end up using ~monthly or so for various things (usually with more switches added as needed):
GNU Parallel as queue system/batch manager
# start queue
true >jobqueue; tail -n+0 -f jobqueue | parallel
# add job
echo my_command my_arg >> jobqueue
# to start queue for remote execution
true >jobqueue; tail -n+0 -f jobqueue | parallel -S ..Before I learned of parallel, I tried a hack where I'd manually assemble jobs into batches, and wait on the batches before starting the next. It achieved very low system utilization, because inevitably, one job each the batch takes much longer than the rest. A slight improvement (still not good), is to use `split` to chop your jobs file into $num_cores chunks, and background each chunk. But still, this gets low utilization. Problem being that you aren't using a thread/worker pool.
Parallel (or, TIL, xargs) can maintain 100% system utilization, until the very last $num_cores jobs.
But it can be done in pure BASH: https://gist.github.com/mped-oticon/b11dafa937e694ce4fa6fbf2...
GNU parallel supports expansion, which bash_parallel doesn't. However bash_parallel works with bash functions, which GNU parallel doesn't.
Nix + GDAL + GNUParallel + autoscaling groups === massive geospatial data processing pipeline
#!/bin/bash
cat - | parallel --line-buffer --pipe --roundrobin jq "$@"
E.g. in Python this would all be very easy to do. Just start a bunch of threads and e.g. invoke subprocess.run() from them.
I am trying to use Python by default when writing scripts nowadays, but sometimes the best tool for the job isn't Python or writing your own Python.
From this perspective, the languages of the glue, the libraries, and the external code all matter less than the ease of writing the glue; interfacing with the external code; and maintaining the libraries. The best language for this probably comes down to a combination of what you're comfortable writing (and reading, and maintaining) and what kinds of tasks you're trying to solve.
For me personally, using Python glue and libraries strikes a pretty good balance here. Writing a script "in Python" doesn't mean you need to reinvent the wheel. If you think `parallel` provides a better interface for map-reduce parallelism than `subprocess` (or than a library function you've written on top of `subprocess`), no problem: you can just call `parallel` from Python (and you'll probably find yourself writing a library function on top of it to abstract away the fact that it's a shell script).
But if you're much more effective working in Bash than Python, then writing your glue and developing your libraries in Bash could be the way to go.
Done that many, many times and honestly combining python with parallel is in many cases the best way to go. Write your python script to be as fast as possible on one core and then use parallel to run it on all your cores. This has the added advantage that you can go from running on all the cores on your machine to running on all the cores on a 100 machine cluster by just changing a couple of lines of code.
Then you _have_ resort to 'wait <pid>' with the 20 lines of bash coded need to manage all those PIDs. I have a large editor bash snippet just for that.
It might be a culture thing. In .NET code I see people running things in parallel a lot within code but maybe this is less so for linux tools.
Maybe functional programming style could lend to a parallel-first programming style, with heuristics to decide when it isn’t worth it.
Imagine a world where there were only GPUs for example - then everyone by default would be running parallel-first code, and in that imaginary world you would need to do nothing to run a series of bash commands piping into each other in parallel.