Kalman filters and more advanced aggregators add non-trivial latency. So even if one does not care about cost, there can be a drawback from having an extra sensor.
Cars and roads are built for human reaction times. That's why you have low speed limits in urban areas. You can have a pile of latencies attributable to processing a scene and still have superhuman reaction time that contributes to outperforming humans.
It's analogous to communications latency. High latencies are very annoying to humans, but below a threshold they stop mattering.
