It's not really about the CPU architecture, but the peripherals of the devices. In the x86 world, the peripherals are governed by the "PC" standard, which these days is implemented by the chipset on your motherboard. They all have similar functionality, although you still do need some chipset-specific drivers in your kernel.
In the ARM world, the landscape is different. There has never been a de facto standard configuration of an ARM computer like the PC was/is for x86. There are ARM computers that are more like a micro controller and there are tablet/mobile devices based on ARM that have a system on chip (SoC) that's got all the peripherals, including GPUs, network adapters, etc. Every SoC is pretty different, there's a bunch of devices connected with i2c, spi and other buses. These are required to control the power and clock management and all the I/O devices.
ARM (the company) doesn't really design these, they just license ARM (the architecture) to a SoC manufacturer and call it a day. This leads to a very diverse and fragmented landscape.
The ARM world has been making strides to reduce the amount of chip-specific code. One good example is the device-tree configuration which, in theory, allows running the same kernel binary on different devices. With device-tree, you pass a config file from the bootloader to the kernel, which contains configuration (i/o memory maps, etc) of the peripherals (which you'd query from the hw/firmware in a x86 machine). This is a huge improvement over what was before it, but there's still a long way to go. And there's still plenty of older ARM SoCs out there that don't support this.
But yes, it's pretty much a mess. There's a lot of duplicate work that every ARM SoC manufacturer has to go through. Given that the mobile world is financed by a planned obsolescence cycle where they expect customers to upgrade every few years, I find it unlikely that this issue will be solved any time soon.
