story
Specifically, we are changing hiring across 3 dimensions: > Tasks: Real-world tasks on code repositories vs standard algorithmic-style puzzles > Evaluation: AI fluency, orchestration skills vs functional correctness > Candidate experience: Agentic IDE vs a simple code editor
In the “old world,” you could ask multiple questions and triangulate skill from answers. Now it seems like evaluation depends heavily on tools and models that keep changing month to month.
So I’m curious: > What signals actually correlate with strong engineers today? > How do you design interviews that don’t become obsolete with the next model release? > Are algorithmic interviews still useful at all?
Would love to hear from people who have recently changed their hiring process or have been interviewed using this new approach.
Several hiring managers in the group went all in on AI-assisted interviews because they wanted the interviews to match the tools that engineers can use on their work. Most of them have gone full circle and returned back to no-AI interviews.
The main problem with AI-assisted interviews is that they become a test of how familiar the candidate is with the specific AI tool you're letting them use. They started getting inverted signals because the hardcore vibecoders knew all the tricks to brute force the problem with high token spend. They'd do things like spend the interview trying to spin up parallel subagents to brute force a solution.
Then the careful coders who tried to understand the problem and do it right were penalized because every minute they spent trying to do the problem (instead of offloading all cognitive load to AI) was time lost to letting AI do the work.
There were also simpler problems like when someone was familiar with some visual LLM interface tool but didn't have a familiar workflow for the CLI tool used in the interview.
Most people went back to coding interviews that forbid AI and test coding skills, combined with a discussion about their AI experience.
My takeaway was that it's easy to teach new hires how to use AI tools on the job, but it's much harder to bring someone with weak coding skills up to the level of someone with strong coding skills. It's even harder when that person is leaning on AI so much that they're not learning how to code anything.
Can’t you just…tell them not do this? Or give them limited model access instead of full Claude Code / Codex?
(1) understanding software engineering (for one thing knowing if answers make sense)
(2) subject matter expertise and the ability to communicate with SMEs, fake being an SME by reading books, see the old "knowledge engineer" construct from the 1980s.
(3) knowing specifics about AI coding.
I think (1) and (2) are 80-90% of what leads to success in the long term. My guess is the models are going to get better so (3) skills have a short half life and will matter less, but (1) and (2) will stay the same.
Maybe I'm cynical but if I was designing screeners for this thing I would ask people things like
"How many accounts do you follow on X about AI?" where the right answer is "I don't have an X account" and the higher the count the worse it is.
"What percent of your programming time do you spend thinking about AI programming tools?" and anything over 20% is suspect (but maybe it is a tooling job or something in which case I'd drop it)
That is, I want to see that somebody used AI tools to deliver something 100% done end-to-end that worked and I'd like to see them spending 80% of their time doing.
I'd also be thinking about screeners designed to detect FOMO attitudes and reject people for it.
The trouble w/ AI slop is that the people who make it don’t know it is AI slop, not that it was AI generated.
* Non-technical recruiters using a list of technical trivia questions and a keywords list to check if you answer correctly (e.g. what is the default isolation level in postgres) - this is the worst.
* Code reviews but on non-generic code, like on a specific framework/product/SDK code in which you may or may not have prior experience - slightly less worse, but if you are going to do this, it is counter intuitive to hire someone who doesn't have production experience in that area.
* take home assignments where they say "we recommend you to not spend more than x hours on this", but then, they expect a very sophisticated work and will reject you if you turn-in a "simple" implementation. Maybe they expect speed and quality?
* leetcode is still a thing.
* non-leetcode, but livecoding, on a likely OOP many to many relationship problem
* give a spec in the first x minutes and leave the interview, you do your thing and then they join in the last y minutes - this is my favorite.
I haven't yet participated in a Agentic AI interview. It is not that common AFAIK.
a coding interview that is objective, repeatable, doesn't put the interviewee under pressure (and doesn't trigger unconscious bias based on accent/appearence)
* take home assigment with minimum requirements criteria and tell them "add a feature of your own" or "any extra work is appreciated".
* Pay them for their time and tokens.
* Use a custom agent to review the code and see how many "high", "medium", "low" issues the agent identifies.
I never finished a take-home assignment within the given time frame. Once I even spent a whole weekend working on a solution to something that was described as a 2h problem. I feel it's just a way for the companies to make it look like it's a small commitment and they are not wasting your time excessively. If you decide to spend 2 days instead of 2 hours then "it's on you" ... But if you don't you'll fail.
It's good starter code for AI-enabled projects that are cloud agnostic and have good security practices :)
It also, probably, got me the job. At this point, that's honestly secondary. I can create new tools way faster!
One failed takehome assignment, years ago, became an amazing portfolio piece for 2 jobs down the line. The reason the original company failed me: not a standard full-stack web app. I broke the client-server model and made a P2P web app (drawing together etc.).
Good developer will give good prompts, because knows what to ask, and what might be the problem. Good developer can read the code and point badly generated one, and learn the AI how to perform better, which style =to follow. Good developer can evaluate if the used algorithm is the proper for the task and give suggestions, if needed. Good developer can optimize token usage, by using scripts, for example.
Yes, skills in prompting, knowing about new tools and how to use them is also mandatory, but not the most important one, in my opinion.
I guess better soft/social skill are needed. Some people just can't express themselves in real world, and probably they will have difficulties expressing themselves in free texts as well
That is, looking at code that's been written by AI and see what's wrong, what's superfluous and what's missing.
For preparing the code for the interview, I would suggest prompting Claude Code using a requirements document that's purposefully a bit vague so that the AI will have to make choices when writing that code.
When you have the interviewee come over, show them the code and have them criticize those choices and edit the code manually (I know) so that they can demonstrate that they can intervene in the AI process at the correct inflection points.
1) Do they understand the ecosystem of algorithms and models? How do those coexist? Statistical models are the right choice sometimes. Sometimes it's XGBoosted RFs, sometimes NNs, LLMs, etc. And they're not mutually exclusive. I don't think that has changed since AI to be honest - though I get bad candidates that say LLM-everything of course.
2) AI-assisted fluency. Not just in coding, but in concept build. I don't expect them to have the velocity of an AI-fluent principal engineer, but I want to see that they're not resistant to AI-assistance. This is obviously new.
3) Experience with production systems, more than before. Pre-AI, I'd accept that a recent grad might be tuned towards models and algorithms, and wouldn't know much about frontend or backend, or anything you run into in production environments. Given the ease with which you can now setup a small DB, your modeling pipeline, and a full react dashboard or fastAPI frontend...I'd at least like to see they've dabbled in all of that, have a rough sense of it. I don't need them to be full-stack, or even comfortable with it - but AI has raised the breadth bar for me.
We are in a new age when the barrier to building is literally the lowest it's ever been. If your GitHub still is a dead graveyard fix that immediately.
I have over 2k commits in 2026 alone and I'm not even an IC anymore. Not including my work account.
I have like 12 side projects being built usually, working on 3-4 in parallel. A few are open source and growing.
All of this has helped me learn at a crazy rate. If your candidate is still just slogging 9-5, doing stand-ups then clocking out without anything extra.... That's all you need to know.
Curious, self learning, high potential people with a track record and good communication skills. That's it.
Now I ask them to code Google Search in 1 hour
If the candidate turns in AI slop and doesn't understand the fundamentals of what they're working on, reject. If they took the time to learn the subject matter and feed it their own ideas to improve the output, awesome.
