If "we'll have too many cars on the freeway following the speed limit" ranks as a serious concern, I think we've really lost the plot.
I recently drove by a fatal accident that had just happened on the freeway. A man on the street had been ripped in half, and his body was lying on the road. I can't imagine the scene is all that unlike the 40 thousand other US road deaths that happen every year.
As a driver I'm willing to accept some minor inconvenience to improve the situation. As a rider I trust Waymo's more than human drivers.