Not always. See ricoh gr. Chunkier than a smartphone with no case? yes. but still fits in the pocket, and probably not much fatter than that same phone after the user has slapped a case and a wallet or a pop socket onto it.

Is that strictly true? If you want it as thin as today's flagship phones, sure, but if you just go for pocketability, it might not be. There are some fantastically compact, perfectly good prime lenses out there. The top end point-and-shoot lines haven't been updated in ages, but even the old ones were are a pretty damned good compromise.