We've already learned this via curb cutouts. An argument like yours was proposed (eg. we don't need cutouts in curbs for wheelchairs, so few people have wheelchairs) but it turns out that everyone benefits from the thing installed (child strollers, luggage pulling, movers, anything on wheels suddenly benefited). The position here is that this is similar.
Having something that is readable to the blind could suddenly benefit a wide, diverse range of people- colorblind folk, people who just underwent eye surgery, cataract sufferers, even photosensitive or light-induced migraine sufferers suddenly benefit! You could even argue that it would help reduce blue light exposure at night if you could browse without turning on your screen and suddenly theres an opportunity for literally everyone using a screen to benefit from ADA compliant websites for the blind.