Great point on this possibility... A big source of brokenness with tech interviews is that so many interviewers want to setup an interview (often adversarial, where the interviewer holds more cards) where candidates get ranked primarily on subjective hidden variables. If the real test isn't the given test, the interviewer sucks. Measure as directly as you can what you care about, whether it's technical criteria or softer criteria, and if the answer is on one candidate's resume but not another's, ask about it rather than assume the other doesn't have it. Want to know how the candidate collaborates? Prime them with that expectation, that the problem solving is a joint effort. Want to know how someone functions in the face of conflict? Ask about their experience with conflict and resolving it, rather than try to setup some conflict and seeing how they respond (or don't). Want to see if they can write a good unit test? Ask them to write some tests rather than saying something vague like "develop as you normally would". Whatever it is, the important part is making clear to the candidate as much as possible what is the hidden state in your head that you're using to evaluate them.
My own dream as candidate/interviewer is to spend some hours trying to solve an interesting problem together that neither of us has solved before. Unfortunately it's not repeatable for the interviewer (at least once it gets solved, until then there might be a way with a big problem to bootstrap a candidate to build on past candidate+interviewer's efforts), and even then its main use is judging "works with the interviewer well" when there are probably better things you should be measuring in most cases.
