That's completely absurd. There could be a 1,000 very legitimate reasons why even a serious or committed candidate will not want to accept an offer: the pay's not right, the role isn't quite right, there's not enough responsibility (or too much) he doesn't think there's sufficient opportunity for career progression, he doesn't like his prospective boss or team, they want you to work in a different location, or travel a lot, or travel too little etc etc etc etc.
A job interview is not a one-way street. It's about discovery and due diligence. It's perfectly acceptable for either side to pull put if they feel that for whatever reason the other side isn't a good fit. And neither side can be sure of a fit until they enter discussions.
You'll just end up shooting yourself in the foot and ditching great candidates: bad for you, and bad for your clients.