In addition to what AnthonyMouse said, it also gets a bit more complicated than that. If you have a high performance game engine, it's very likely that the inner loops are written in assembly to squeeze every last drop of performance out. The guys who write this code are optimizing for the OS's memory handling behaviors, aligning data to fit into cache lines, using SSE instructions for faster matrix multiplications, and all sorts of other exotic techniques that "mere mortals" don't even think about.
So when you change platforms from, say, the Xbox 360 to the PS3, those assumptions you made about the OS, the CPU, the chipset, and other minute details are suddenly completely, horribly wrong. So the guy who wrote all of that highly optimized code has to write it again to optimize against a completely different set of quirks.