I've honestly never had a problem cross-compiling for the Raspberry Pi. I learned my lesson back when it first came out; I needed a new ntpd and it took 24 hours to build on the Pi. I then realized the settings were wrong and spent about 20 minutes setting up the cross-compilation machinery on my x86 Linux box instead of waiting another day. "apt-get install crossbuild-essential-armhf" and some configuration and your build is done in seconds. Well worth it.

Modern languages are even easier. I can build a Go binary for the Raspberry Pi by setting one or two environment variables; "GOARCH=arm GOOS=linux go build ./whatever". Wonderful.

The Raspberry Pi has improved since the original. I recently needed llvm compiled from source and it only took on the order of hours on a Pi 4. (GCC was unusable, though; uses too much memory. Had to use clang from the package manager to build a new LLVM. The efficiency was impressive.)