Users that want the choice can choose. Devs that want the choice can choose. You can stay in your walled garden. And considering the uptake of alternate app stores on Android this seems like FUD to me.
If they had the option of going to a less restrictive store, they probably would, and they would do things I am not happy with there. That would be a loss for me.
I think you’re underestimating the power of defaults. App developers know that the vast majority of users will not install an alternative app store, even if the platform allows them to. For proof, see the Android app ecosystem. There has been no migration from Google’s Play Store. There isn’t even a hint of store fragmentation. What you see is: 1) some OEMs run their own stores, which mostly rehost apps from Google’s store. These stores are not meant to be installed by arbitrary Android devices, so there is no chance that a user might feel compelled to install the store on their device. 2) F-Droid exists to host FOSS apps, some of which violate some asinine Google policy and as such are exclusive, but most of which are also available on the Play Store.
Basically, we have strong real-world evidence that allowing sideloading does not create app store fragmentation. If you want to argue against sideloading from a security standpoint, you’d be better served with the “sideloading allows an abusive spouse to install a keylogger/tracker on their partner’s phone” narrative.