Small phones are also difficult. Memory, processors, and batteries don't shrink. For an iPhone mini, they're going to be shipping essentially the same chips taking up the same amount of space. That space is going to have to come at the expense of things like the battery and cooling. It's a lot easier to engineer something with looser tolerances. If you have a giant phone, it's easy to have extra room to keep the phone cool and stuff in a battery.
It also probably meant limiting some choices for the rest of the iPhone lineup. Apple wants to be able to re-use components and to some extent it's going to mean that Apple either has to make choices that work for both a 6.1" and 5.4" form factor or do separate things.
There is some demand for an iPhone mini. I love the iPhone mini. I also see the challenge for Apple.
I think there's also a reason why we haven't seen a successful Android mini phone. It's hard to make a mini phone and the sales numbers are comparatively small.
But maybe we'll see an iPhone mini in a few years time. If Apple can create an integrated CPU/modem/WiFi/Bluetooth chip, that could end up saving a decent amount of space while also reducing power requirements. Maybe we'll be able to go SIM-less around the world and that could save space.
At the same time, it's hard to make the same number of people make another, more challenging form factor and it's hard to scale out with more people too. Plus, do you put your best engineers on the hardest project (the mini) when it's only 3% of sales? Or do you hire new, less experienced, possibly lower skilled people for that and hope you don't put out a product that isn't good?
It's a tough challenge for a tiny amount of sales which, ultimately, aren't going to decide to leave Apple for Android where they also can't get a small phone.