It’s way more complex than that. Your brain is running pattern recognition on the persons height, gender, age, race, clothing, gait, where they are looking, etc to work out the risk level.

If you saw an old lady carrying shopping bags you’d feel no risk of danger vs a large male in a hoodie walking in a weird way and looking at you.