people make this mistake over and over again. there's no point in taking anything but the best time when talking about performance in a preemptive multiprocessing environment. the best time isn't an outlier, every time other than it is!
Benchmarks are only useful if they're indicative of what the likely outcome in a real-world scenario will be.
What you propose is much akin to those completely unrealistic "The Computer Language Benchmarks Game"-style benchmarks that show JavaScript (or Java, or PHP, or whichever slower language you prefer to use instead) as being as fast as C or C++. Perhaps that's true, but only when you discard reality almost completely. JavaScript only ends up performing decently because the benchmark is trivial, and the implementation has been highly tuned, often into a form that we'd never actually see in any real-world product. And predictably, JavaScript's performance ends up being much worse than C's or C++'s when it comes to real software, contrary to what the flawed benchmark suggests.
This isn't a practical benchmark, though. This is an experimental performance comparison. So to ensure that your results aren't influenced by other variables, it's a good idea to attempt to produce the ideal environment for best performance across both tests. So binding cores(to eliminate as much variance from Linux's thread scheduler) and running nothing else on the machine are actually good decisions to make here.
Comparisons to the Computer Language Benchmarks Game is somewhat irrelevant here, because the comparison being run is actually extremely simple: Using a GCC executable compiled as C versus a GCC executable compiled as C++. The traditional failings of the game aren't here(implementation issues, language-specific tuning issues, etc.) because the code that GCC is compiled from is identical, the compiler options should be identical(except for C vs C++ compile setting), and the test setup should be run on the same machine, with the same compiler options and tuning choices.
I've learned that it's unrealistic to expect someone to have actually looked at "The Computer Language Benchmarks Game" before pointing to it as an example for whatever point they wish to make ;-)
The only mystery here is why you think we wouldn't notice that "the implementation has been highly tuned" etc applies just as much to the C and C++ programs as to the JavaScript programs.
- GCC recently switched from building their own code as C, to building as C++. At the moment they only use the subset of C++ which is C, but this affects compiler behaviour.
- People complained that C++ programs couldn't be compiled as efficiently as C programs, and this would make things slower.
- This post shows that compiling GCC as C, versus C++, does not result in an appreciable performance change. He benchmarks this by building the kernel using both versions of GCC.
This can look like nitpicking, but it's really annoying to again and again hear such things from mouth of C++ gurus.
Now give it to me ;-)
I can see how this is a little confusing but it shouldn't be that hard. And yet there have been hours of comments on this story that miss this point.
Nice to see him here - "ciao" from Italy:-)
Also, he used to poke fun at C++ (many older C devs did that) but more and more of them are coming around to C++. The last automobile systems programmers I worked with were switching to C++ as well. In ten years, C++ will be the new C.
But I like measurements, even if they don't confirm my baises.
Saying that, 40 isn't very many times to draw statistical conclusions so you'd probably need more data.
This wasn't testing the speed of compiling a program as C++ versus compiling it as C; it was testing the speed of compiling the Linux kernel using GCC-compiled-as-C++ as the compiler versus using GCC-compiled-as-C as the compiler.
(And it concluded that the difference was insignificant anyway!)
What would be interesting is how long it now takes to compile GCC itself. For code that does not use C++ features, I also expect a very minor difference.
