It actually scales better the more source files you have.
The bottleneck currently is link job dependencies.
LLVM is a good example of that, It has 2000 tasks to complete. It hits a point about 360 in where it has to link a tool that is used to generate code for a later stage.
The fastest build would be all source files with no linking. That could be done in about 30 seconds.
Same would apply if you had 1k or 10k source files, 30 seconds.
I picked LLVM because it is the toughest benchmarks for this because I want real world use cases and expectations.
The demo shows an 8 core machine. On a 16 core machine it can build it in under 2 minutes.
https://asciinema.org/a/214eea3mnjcn1jeiuu3s3qwso