Isn't the job of your Makefile to require recompilation of only those source files that have changed?
I just did a random google of "how long to compile linux kernel" and came across this: https://plus.google.com/u/0/+LinusTorvalds/posts/6BxnSisp8fU
One nice excerpt:
"Total build time after make clean is about 1min, give or take 10secs. `touch include/linux/version.h` is 6 seconds to rebuild. Just doing a rebuild without touching anything is 2.7secs.
For a defconfig, it does build a godawful amount of modules :^)
'allnoconfig; make' is 16 seconds..."