I wouldn't worry myself with that. Since a black hole behaves exactly like a normal star at a large distance, the chance is very small, even if there are many of them. For example, when our galaxy merges with Andromeda there will be only ~1 star collision.
Normal black holes are really small, much smaller than a planet, which we can barely detect now. And those planets must be orbiting a star and in the correct inclination to be detected.