I currently do something pretty similar by using the perf subsystem in the Linux kernel to track the behavior of each benchmark function. In my early measurements I found concurrent benchmarking to introduce unacceptable noise even with this measurement tool and with cgroups/cpusets used to pin the different processes to their own cores. Instead of trying to tune the system to account for this, I chose to build tooling for managing a single runner per small cheap machine.

Please don't, because memory access pattern will be very different.

Some of those target benchmarks are on Rayon, and we've found that valgrind interferes with threading way too much to be useful there.

This is one of the many metrics of the official Rust compiler performance benchmarks [1].

[1]: https://perf.rust-lang.org/nll-dashboard.html

I haven't used callgrind, but wouldn't running benchmarks concurrently still lead to cache interference?