Partly because, historically (e.g. on feature phones), the GSM baseband serves as the primary processor (with real-time responsiveness requirements), with "applications" as a subordinate function. The concept of GSM being "just" a modem peripheral is a more recent development, coming more from the laptop arena; pushing that model down into phones (especially
cheap phones) will take work.
Even on top of that, the concept of not trusting your peripherals is a recent one as well. Ideally, all hardware peripherals would have no more permissions than they need; for instance, no ability to DMA except to specific pre-arranged regions. In practice, most systems don't actually set up that level of security.