How do we know? I haven't heard of any studies that compare people's first impressions against the person's actual disposition or personality.
For example, this article pointed out that honesty was a trait people tend to read from faces, but it didn't ask the question: are people's reading accurate? It assumes that an honest-looking face has no bearing on a person's actual integrity, but maybe it does. We haven't actually measured.
To be clear: I'm not trying to agree or disagree with you, just point out that we don't have any strong evidence either way. As far as I know both positions are unsupported scientifically.
