My theory is that much of this effect is an error, or at least a far-less-than-ideal effort, on the part of the designers. Of course it’s hard to sell a low-end “mini” device with a worse camera, worse battery life, etc. But that’s not actually what I, or many people I discuss this with, want. I would happily buy a premium device that is short and narrow, and possibly even thicker as a tradeoff. There’s plenty of unexplored room in the design space here. For example: start with an iPhone Pro or whatever the Android equivalent du jour is. Keep the camera unchanged. Shrink the display but keep the same quality (at least equal pixel density). Now puff out the back so that the camera lenses are flat or even slightly recessed. Use the resulting added volume to compensate for the decrease in volume due to decreasing the other dimensions. Market the think as a Whatever Phone Pro Compact, and advertise clearly that the battery life is every bit as good as the non-Compact model version. Show off cool pictures models sticking this thing in their cool jeans pockets without them sticking out. Charge the same price as the ordinary Pro model.
As far as I know, no one has tried anything like this in recent memory. The iPhone 12 and 13 Mini were always marketed as the cheaper versions, and the cute little old SE model was very much a low-end version. Last I checked, there was no 5G Android device with similar dimensions from any manufacturer.
