It seems you're saying that the 'fitness algorithm' filters randomness and therefore the result is 'less random.' And while I can see your point, I'm of the mind that filtered noise is just less noise. But it's still noise.
If I give you a stream of decimal digits, and you filter out everything except for 9s, would you call the resulting stream as random as the input stream?
How about if you filter every digit that is not exactly one more than the last unfiltered digit?