Yeah I figured there might be a rub somewhere otherwise it would already be a thing, but since it's technically an ARM it sounded vaguely promising. What about a 32 bit build? I think those used to be able to work with without virtual addresses.
aha, and this reminds me about μClinux [1] which targets microcontrollers without a MMU. I installing it on 2005 iPod Classic 5G, and was able to then put a gameboy emulator on it.
Though μClinux project seems dead, however the key component of that is a ELF to bFLT (binary flat) converter [1] for no-mmu Linux targets, which is alive on github [2].