Heck, why have driveways? In cities, at least, I could fully imagine new townhouse developments operating more like apartment buildings: one central parking lot for the "pool" of (mostly) driverless cars. Your car drops you off at your door and then parks in the pool; then comes back out to get you in the morning.
* A large percentage of people will want to have their own car because they don't want to put up with the occasional hassle of unavailability of rental cars during a peak time, and/or occasional long waits for the car to reach them (particularly when they are being picked up in un-dense areas). And before you hand-wave this away, note that the more of a cost advantage you give to rental cars (ie, the higher the utilization of those cars), the more of a disadvantage cost/wait time will be.
* People -- and very especially families with children -- also value cars as a mobile depot for their stuff. If you have kids of certain ages, this is practically a necessity. I definitely am not going to screw around with not just a carseat but like a dozen books and snacks and wipes and cups and so forth every time I get into/out of a rental car.
* Then cleaning and status and all the other things that people regularly bring up here.
* To address the mobile depot use case, I've heard of proposals to have storage areas (possibly trunk, but not necessarily) that are only accessible by the owner, not by renters. Even that may not be perfect though, I do see your point.
Is there a sweet spot where you still experience significant cost reductions but the inconvenience is small enough that you don't mind it? Well, certainly, for some people! But I think the number of people is going to be a lot smaller than many commentators think.
(The only cases I see where there's no increase in the rides-for-hire business after full automation is if the cost of cars drops to the point where there's very little incentive to try to reduce the costs for most people. Given that cars are quite expensive now, that seems unlikely. I can construct a scenario, but you basically have to believe that, first, the additional cost for driverless sensors+software is very low, second, that driverless software is so safe that you can drastically decrease physical safety features on cars, and third, probably that automation drives significant GDP growth).
It's not that hard. For one thing, you don't need to increase the price much, you just need to increase it a little so that you push cost-over-everything riders out of the queue. There are enough of those that whatever capacity is left should be plenty.
And you are worried those riders who were pushed out of the queue will go back to car ownership, but they are by definition cost-over-everything people, so of course they won't do that. If they could afford to own a car they could afford peak pricing.
My point is that millions of them already exist, with a distributed network of roads leading to them, as well as storage (garages). It would be costly to suddenly rip all those up.