I bet part of the reason for that is because of how it's presented. Google puts up several warnings that you have to click through and they're all along the lines of "this will
ruin you phone!" IMO having sideloading isn't anything more than a well thought out, long term plan to deflect accusations of anti-competitive behavior. In reality they do everything they can to discourage users from sideloading.
The biggest issue for me personally is that it's not clear how, or if, sideloaded apps are sandboxed and how they're granted permissions. The warnings seem intentionally designed to play on those fears. If they're sandboxed the same as other apps and have fine grained permission controls I'd do a lot more sideloading.
