The SE is the previous iPhone design. Before that happened to be the iPhone 5/S, now it's the iPhone 6/7/8 design. There are likely a lot of people with iPhone 6, 6S, and 7 that might upgrade but don't want to spend $700+ and lose the home button.
I don't think the SE was targeted at small phone lovers, it just happened to be that since that's the manufacturing and design they had already created for the 5 and 5S.
Rumor is that the iPhone 12 this fall will include a smaller 5.4" model. This would actually be smaller than the 4.7" SE given that the screen goes to the edges. That's likely the "premium small phone" replacement, not this new 4.7" SE.
https://imgix.bustle.com/inverse/17/df/7b/88/27fe/4afe/aff4/...