Any decently sized C++ or Swift code base is going to have compile times way over 5s. Many just moderately sized projects have build times over 10 minutes.
For moderate to large projects, link time can be a huge factor even for incremental builds. It might be that faster IO is the better investment, though.