Because the deaths will be network level effects that a person would be helpless to mitigate through behavior. There's no sense of being a careful or responsible driver with a self-driving car; its all whether or not the algorithm or software is correct.
It's a different type of potential error and a much more scary one. I can mitigate human drivers as a pedestrian by taking care walking, but I cannot mitigate an AI that thinks my shirt makes me look invisible due to its learning being deceived by a pattern.