We can't fully trust the correctness of modern complicated CPU designs, leading to problems like <insert all speculative bypasses that have affected Intel CPUs the past 2 years>. But despite their complexity, CPUs and the CPU part of a smartphone SoC are usually extremely well understood (relatively speaking). The reason is that you actually need to run your software on these CPUs, so they need to be understood rather well. With better understanding comes better trust.

On the other hand, the baseband processor is mostly unknown, black box hardware, running unknown black box software, that completely controls the transmission of cellular data. Of course it would be horrible if there was no separation between the CPU and baseband. You shouldn't trust that setup. But as it turns out, separation does exist!