I think this is a huge mistake. I work in data science, and nearly universally my colleagues are motivated by opportunities to learn. If you're testing for the very rare day, so that your employee has the right answer in their front pocket when that happens they are never going to feel challenged, feel like they're growing, and they're going to jump ship.

I think you're doing snobby filtering, and it's not to your benefit.

There are a few problems with this. If your "hardest day" deviates far from your "average day", while being very infrequent, you're still not doing a good job enticing the candidate. Worse, by looking for people who have already acquired the skills to combat the worst you can throw at them, you're severely stagnating their growth.

Then comes the cherry on top, if your company is doing it while not providing absolutely stellar benefits to function as external motivation, you're further incentivizing people to work on ditching your place. Many companies do not give total comps in line with the skill level tested for, setting themselves up for failure in this regard. Altogether, you're still looking at this primarily from the job interviewer's perspective, not the perspective of the candidate which translates into the company's long term (lack of) health.

>you will eventually have a problem that nobody can solve

No. Someone has to be the first person to solve a question. This strategy doesn't take into account the capacity of individuals to learn and overcome challenges. What's more, most individuals will not be on the highest level, allowing mentors to train them to higher levels.

>I don’t necessarily disagree

It's not about disagreement. Schmidt and Hunter did a meta-analysis on this way back in 1998, and psychology has yet to find the silver bullet. Look at table 1[1]. Most studies checking the market find similar things. It is ridiculously difficult to find correlations between job performance and selection criteria. All corporates, including FAANG, are not exempt from making the very human mistake of doing something which seems great, but ends up being part of survivorship bias. In something as multi-faceted as the job market, combined with the young and everchanging nature of software development, making statements on correlations shouldn't be taken for granted.

Notice how the study above points out years of experience as one of the worst predictors. Most of the job market still uses YoE as if it is gospel. The market is nowhere near as rational as people like to believe it is.

[1]: https://www.researchgate.net/publication/232564809_The_Valid...