I don't think I ever rebuilt the entire kernel when writing Linux drivers. I dimly recall one instance when I was dabbling with/hooking through the network stack and had to do it for some reason, but I don't recall the reason. Then again, I didn't develop drivers for pre-2.6 kernels, so I may be missing something specific to your codebase.
> I used to joke "Linux will never get widespread until they have a Have Disk button" meaning I can get a driver from a vendor with a push of a button during installation.
The chasm between communities is remarkable :). At the time, I liked Linux for precisely the opposite thing: it came with all the drivers I needed out of the box. No fiddling with CDs and floppies after installation. Except for the odd device that required me to download a driver from the vendor's website (and even then I didn't need to reboot my computer, at least!) I was good to go. Whereas, after installing Windows on a PC, the first thing to do was spend an hour or so installing drivers for all that stuff that made it possible to have I/O more advanced than 640x480, 16 colors and PC speaker.
Granted, though, my experience then was limited to desktop computers. Linux hardware support was far less diverse than it is today. There was no fiddling to do if you had stuff that was supported; but the chances of it being supported weren't always good.