It does significantly speed up clean builds in many cases, to the point that those clean builds can even be faster than many "incremental" builds in some cases.
I do not know of any compiler that can compile different parts of a translation unit in parallel. But plenty of systems (including the venerable `make -j`) can compile separate translation units in parallel because they are independent.
You don't want to clear caches for release builds. That just makes release builds unnecessarily slow and impedes the flow of Continuous Deployment. You just need to have separate caches for different build flags, including a dedicated cache for release builds.
