Agreed, building on the typical target is a terrible idea. None of my suggestions are to build on target, but rather specific units on some capable hardware of the same architecture. The main build will still run on your own workstation of any architecture. Only specific units get farmed out to another machine, but worst case can be run in QEMU user mode emulation if there is no native machine runner available.

This remote build runner is not implemented yet, just ideas.