honestly that is something I value as well, but I think that's when a lot of bias can come into play. Perhaps that bias is desired for culture fitting, but you still have to be careful if protected-class stuff comes up and you want to reject the person.
fwiw, when I interview candidates in a technical interview, I still attempt at some small talk to get a sense of "can I have a normal conversation with this person"
(appreciate the sensible responses btw!)