"at least cache the latest successful build from a package version"
Tools like guix and nix aim to describe packages in such a way that for the same input (package dependencies) you get the same output.
If the output is a function of the input, and you know a cached build of a package was built with the same input, then using the cached build of a package would be the same as if you'd built it yourself.
It does depend on whether the packages are written in a "pure" way. (Or "impure", such that building with the same inputs might result in different outputs).
https://github.com/NixOS/nix/issues/2270