The entire docks, of a port city, ie that's the only reason people built a city there, is restricted.
The University has a department with its buildings inside the restricted zone (Oceanography, it would be stupid to not put it next to the docks).
Do I expect the average driver to be in and out every five minutes? No. Do I expect taxi drivers to have seen the docks before and know that, duh, "I have a fare to pick up" is a perfectly acceptable reason? Yes. That'll be what the guy who did pick us up thought too. How do you think we got in to visit a ship in the docks in the first place? Taxi.
The docks are big. They're docks! We didn't walk out of the docks to meet that taxi, that would take ages, we just went from the car park next to the ship to a road that the taxi driver could find on his map. Unlike Uber, when he couldn't find us I just talked to him on my phone.
Know what else is in the docks? Cruise liners. Need a taxi to the airport after your cruise? Those taxis are coming into the restricted access area. Know what else is restricted? The airport! I wonder if any taxi drivers ever visit the airport...