However, one thing that I haven't seen discussed anywhere, but that strikes me as a hard problem: by having a non-human driver, you're literally taking out the human element of communication.
Everybody is taught about the importance of eye contact in driver's ed. What will we replace it with? I have no clue, and I doubt anyone else does. Yet it's a crucial technique we all use to negotiate traffic every day, regardless of our mode of transportation.
This is a hard problem, and it's less technical than cultural. It will be a bitch to solve.