Yeah, it's basically tradeoffs all around.
The benefit of this approach is it's a lot easier to make sure dependencies end up on the build node so you aren't redownloading and caching the same dependency for multiple artifacts. But then you don't get to take advantage of docker build caching to speed up things when something doesn't change.
That's the part about docker I don't love. I get why it's this way, but I wish there was a better way to have it reuse files between images. The best you can do is a cache mount. But that can run into size issues as time goes on which is annoying.