The only semi-sane workaround I know of is to use pip's `--only-binary` option to prevent any automatic builds at package installation time. Then you usually will also need your own package server for storing precompiled wheels for those third-party-dependencies that don't published compiled packages on pypi. This way you build the packages using some random tool version only once, so if it works the first time it will keep working.
In other language ecosystems were the package management isn't built on a tower of shit, this problem doesn't exist in the first place.