You can't just run over a pedestrian because they use a crosswalk when the signal is telling them to stop. That's not how society works.

These self-driving cars are going to need to be very careful around people if they operate in cities, but if they are careful enough to be safe, they'll also have to deal with groups of bored teenagers herding the cars around for fun.

Sadly, any rule that can't bend to the fact that people are people is doomed to fail.