Location indoors is made possible by sending a list of visible WiFi AP MAC addresses and relative signal strengths to a web service. WiFi MAC addresses are globally unique serial numbers, and Google/Apple maintain huge MAC-to-location databases. Enabling location doesn't just turn on GPS, it sends your location to Apple/Google constantly to retrieve the lat/lon from visible WiFi triangulation.

The only way to keep your mobile location halfway private (your carrier still gets it, and leaks it to your national military) is to keep all location services disabled on the device. Otherwise, simply enabling location access at all gives Google and Apple the aforementioned "free for all access".

I agree that it shouldn't be this way, but it is today.