"You have ... people who want them to fit in your hand"
Are incorrect. The number of people who will actually buy small devices is ... small. The number of people who are so interested in small devices they'll overlook things like a lower battery life and whatever other compromises are needed to achieve the smaller size, likely even fewer.
It's not like it hasn't been tried in the past, people in this thread talk about iPhone minis disappearing - Apple couldn't make them a success. Sony couldn't make them a success either and stopped making them AFAICT. As a market segment you're too small to warrant the investment in designing a small flagship. And if nobody's investing in a small flagship, small midmarket isn't going to happen either.
There do appear to be niche manufacturers in this segment (take a look at https://www.reddit.com/r/smallphones/). If the untapped demand is so huge, I would expect to see them become much more mainstream over time.