Agree that a 'gut feeling' that something is off is not to be ignored; it's likely just factors unconsciously processed.
That said, how much of your 'Strong Yes' assessments are on your gut and how much on you have enough info to see that this candidate is qualified, enthusiastic, and a good fit? Also, how many times have those failed to work out and how many have 'Weak Yes' candidates actually worked out in the end? Those counter-examples might be worth looking into...
