> You don't take someone who has PyCon on their CV and get them to do fizzbuzz on a whiteboard
Disagree. I’ve also interviewed hundreds of candidates over the last couple decades, spanning a half dozen or so companies, and there’s always been a rule of consistency. Giving candidates wildly different interview experiences introduces bias, at the very least, but it also makes it very difficult to select the best candidate among a pool. There is simply no way to compare each and stack rank them, you might as well pick one at random.