Each approach I tried failed because of some limitation that made the whole thing awkward or pointless.Could you explain the limitations that you've run into? I have been using shell.nix + direnv + some caching for 1.5 years now and I have been very happy with it.
The only thing I disliked is that shell.nix is underspecified in the sense that it by default relies on your configured nixpkgs channel. So, I am now using niv to pin down nixpkgs and other package sets (mostly my own) to specific revisions. Once Nix flakes become mainline, this can be done easily without third-party tools.