At a client they use an obscure system called e2factory to build images and that one does model dependencies correctly. It creates a little chroot environment for every package build and errs on the side of rebuilding if in doubt. It's probably less flexible than Bitbake with Yocto, but incremental builds totally work. You might need some all-encompassing integration to get more fine-grained dependencies, but tracking what goes into a build is possible with standard build systems (mainly autoconf / make and CMake / Ninja in this case).