Wouldn't it be cleaner, easier and simpler to keep those separate and add them later as "modules"?
Isn't this how we're supposed to write other types of software? Why is this case different?
The modules included in the kernel are updated whenever the API changes.
For the external modules, every time when a new kernel version is released, they may become broken and there is a lot of work to identify how to update the modules, unless one watches the kernel mail lists every day, because there is also no documentation about how to migrate the old modules to the new kernel.
The reasons for module breakage may be just the movement of some definitions from one header to another, which break compiling, but most frequently some structures gain new members or lose old members, or some functions gain new arguments or lose old arguments.
When there are new members or arguments it is hard to discover what values should be put in them, while when members or arguments are deleted it is hard to discover whether their absence must be compensated somehow, e.g. by inserting invocations to other functions.
In the worst case everything can be solved by reading the kernel source, but that takes a lot of time and those who maintain out-of-kernel modules usually do not do this as a full-time job, so they do not have time to scan every day the kernel mail lists, to see if anyone has plans to make changes that will break their modules.
But has there ever been any attempt to create an abstraction layer in the kernel that does provide a stable ABI? Something that could be used for certain classes of driver?
If you're asking why so many drivers are included in Linux's mainline repository and not maintained off-tree - the major advantage of being upstreamed is that changes to the kernel have to take your driver into account as well. e.g. anyone who changes any APIs will have to update your driver as well.
Moreover, being upstreamed certifies that your driver has gone through a non-trivial review process to be merged into the mainline and is an indication of quality/stability.
The situation of GPL'd drivers in a kernel with an unstable ABI wasn't really planned, but for users it's probably the best outcome.
When only the ABI changes, a recompilation of all sources is enough.
When the API changes, then programmers must manually edit the sources and change function invocations and data structures, to match the new API.
That is exactly what Linux does. Part of building a Linux distribution is determining which modules to ship by default to your users. To the best of my knowledge, the overwhelming majority of drivers are not installed by default, and the average user will never need them.