While "building it backwards" sounds simple to us, it's not simple at all in evolutionary space. Evolution operates over a fitness landscape, where every configuration of an organism has some fitness (essentially, probability of reproducing). It can make good progress as long as there are paths that don't require traversing areas of impractically low fitness. The problem in this case is that between "normal" and "mirror" is an area of absurdly low fitness, which evolution shouldn't be expected to be able to cross.