Absolutely not true. The problem is not the compilation of one single file, but that every one of these single files pulls in large amounts of headers, distributed over various libraries (e. g. Qt/Boost/STL), all of which won't fit into the disk cache.

If it doesn't make a difference, all that means is that your project is small, or doesn't have too many dependencies. Good for you. But that's not the reality for all projects.