Low ratings are not the only reason that could happen. Uber drivers run businesses. They are not universal service taxis (which often flout the universal service rule, to be true).

Your request for service includes several non-price elements to it, one of which is your perceived likelihood to be a hassle (and reflected in your star rating), one is the amount of unpaid travel required to get to you, and one is what the driver perceives as likely to be your destination. All else being equal, drivers would prefer to end up in portions of the city with a high density of quality fares (say, an airport, the startup district(s) in SF, etc) as opposed to portions of the city with a low density of quality fares (say, the outer edge of the service area).

If drivers make a habit out of taking "bad" trips, their Uber business gets meaningfully less lucrative. Accordingly, expect them to actively avoid "bad" trips. If one's travel patterns routinely result in drivers perceiving you to require "bad" trips, one is going to have a far different experience with Uber relative to a high-frequency no-hassle business traveler.