As for routes only having 4 stops - sure, but that's after you chosen which users will drive in the same car/trip, which is hard in itself.
BTW the dial-a-ride problem is quite similar to the ridesharing problem, and the complexity there is O(number_of_pickup_and_drop_points^2 ) [1]
[1]https://www.itu.dk/people/pagh/CAOS/DARP.pdf - altough it's a bit old, so maybe results have improved since.