Most men are decent, but plenty of men are aggressive, scary, or make women very uncomfortable.
If you ask someone out respectfully, of course that's fine. If you're not taking no for an answer, being hostile, or making someone feel very uncomfortable/unsafe, then you're not doing the right thing.