Those extra journeys can be minimized with decent routing software. Even if it's as simple as ending up at B, then going directly to the nearest customer waiting for a ride.
And fwiw, if Tesla gets their robotaxi stuff working, their 22-passenger "robovan" is basically a bus anyway, just without a driver.