It's how sexual selection works in general throughout the animal kingdom. You can teach yourself to behave otherwise, but you're hardwired to behave a certain way.
Humans exist within a fitness gradient and we naturally evaluate other humans according to how they fit within it.
I don't like it, but we're bloody animals.