Not all environments are as friendly. Downtown San Francisco will be extremely hard to get right -- note that I say get right, not dump a robot car there and hope for the best. There are going to be a lot of crushed pedestrians and damaged parked cars, and I'm not sure the people who live there will value having autonomous vehicles enough to overlook that series of incidents. Then there are suburban and rural environments with their own challenges: poor mapping, roads that aren't obviously roads, one-lane roads, poor visibility, lack of signage, and so on. Not every location has weather like the Valley, either; suppose an autonomous car gets stuck in snow. A human driver can usually get out, possibly with the aid of a shovel, carpet, sand, etc. The robot can't do those things, and the passengers may not be able to (for example, it's unlikely that a robotic taxicab would have any provision for passengers to control it).
The world is a much bigger place than the Valley, and comes with much bigger challenges. That's why I don't see any real likelihood that fully autonomous vehicles will be developed any time soon. It's easy to get lost in narrow thinking when your entire world consists of a single place and a bunch of people all living in that same place practicing groupthink. This problem is much more difficult than you imagine.