If you want to have a large pool of generic hosts that can run any image, then you've just reduced the size of the pool for some images.

If you choose instead to make every secret available to every host, then you've just spread your individual application secrets across many hosts.