Each of those reads may only be 1ms, which would translate into only 100 connections needed.
So the workload you describe — without more detail — may in fact be just fine with that connection count.
But plenty of people also run Postgres with well above 500 max connections.