"The problem is, we're not hiring people to build to a spec. We want to assess your decisions and ability to go from a basic problem to a first implementation"

You could have told the candidate that, then. That would have been some useful information to have for the hours the candidate put into their submission. Or like "hey, we're going to just skim over this with a subjective list of check boxes".

I have enough experience that I probably would have smelled this before I got that deep into the process and noped out, but for the hopeful mid-level/new-senior devs this sounds pretty demoralizing.

Submissions are supposed to predict and handle issues that you guys have had in production? That doesn't seem very reasonable for a 2 hour task.

I wonder if your hiring process is skewed by the perfectionists who spend tens of hours of their submissions. Arguably that's the type of thing you guys would want to select for but it's not at all fair for those who timebox themselves to 2 hours or whatever.

> The submissions that we continue with check a bunch of boxes. The submissions that we pass on don't.

What detail is shared about what "checkboxes" the candidate met and didn't meet? From the parent post and your reply, it doesn't sound like much at all?

Woah, I've been a hiring manager long enough that it's been a while since I've done a take-home code exam myself but I don't even think I would grade a LLM that way (because I moved to AI now). Unless your code exam is super trivial or the boxes themselves are table stakes ("code runs without errors", "code includes more than one function"), coding is creative enough that it's hard to come up with 20 checkboxes that cover whether a sample is any "good" let alone "shows better decision making".

I've had a few bad experiences when sharing feedback with candidates myself and I would understand doing the checkbox approach for feedback and/or just never sending detailed feedback, but actually grading submissions pass/fail based on a subset of criteria you jealously guard from candidates essentially selects for lucky people. If I wanted to do that, I'd just shuffle the submissions by number of bytes and discard everything that's a multiple of 5 or something.

> Your point about valuing peoples' time is important, though. We have not yet found the right balance for everyone.

Did you try paying candidates for the home assessment? People's time costs money. Paying people not only helps attract candidates but also helps the company reduce the list of candidates to take the assessment.

I'm inclined to say the interview process is imbalanced with more power in the hands of the employer. If you're looking for the right balance, try turning over some power from the employer to the candidate.

The candidate's cost was 2-hours worth of time. Was the employer's cost 2-hours worth of time?

Presumably you can't tell if the applicant spent 2h or 6h, and in practice the 6h solution is probably going to look much better. So sure, in theory you may be right, but in practice I bet you select for people who spent (much) more than 2 hours.

And for every company that means it there are five others want to find people that will sink massive amounts of time into these so they can do the same once they start.