We compiled a library of realistic engineering take-home tests and ranked them (opens in new tab)

I recently interviewed with Ramp and enjoyed their model of practical problems, like a take home, but condensed into just one hour with less of the end-to-end expectations of a true take home.

"This article summarizes the practical and theoretical implications of 85 years of research in personnel selection. On the basis of meta-analytic findings, this article presents the validity of 19 selection procedures for predicting job performance and training performance and the validity of paired combinations of general mental ability (GMA) and the 18 other selection procedures. Overall, the 3 combinations with the highest multivariate validity and utility for job performance were GMA plus a work sample test (mean validity of .63), GMA plus an integrity test (mean validity of .65), and GMA plus a structured interview (mean validity of .63)"

1. The research is dated (1998). Long before many of the current best practices in SW Eng were established.

2. Says it is based on 85 years of research. Obviously not IT-related then.

3. Even if we get past that it gives 3 almost equally good methods of hiring where the highest one is GMA and integrity test - not work sample test.

4. Even if we get past all that work sample means work sample. It does not have to be produced under pressure in a weekend as unpaid work which as any professional knows is very hard to bring yourself to do right (being a professional means getting paid for my services as I live from selling them). It can very well be some past work on github etc.

So, unless there is a better/more focused on IT/up to date research proving that take home tests (which btw can be offshored/gamed very easily) lead to better hires I remain highly skeptical of all that and big fan of whiteboard/pair programming.

Lots of the comments are pointing out a 2-4h takehome is a poor value prop from the candidate's side compared to a 1h leetcode which transfers better across companies. (I'd agree with this.) But a 1h leetcode is usually just a first hurdle to what's often another 1-3 rounds, while for me the point of the 2-4h takehome is that it's the only thing. You do it, we grade/talk about, we can arrange a more formal team meeting if you want but that's the whole process.

Am I alone in wanting shorter interviews but longer tests?

I'm not a marketing guy (by any stretch) so maybe my opinion isn't worth much, but I think this is outstanding - provide something already useful, and then invite people to purchase your customized even more useful stuff.

It's always nice to see something well done — kudos!

(You seem to be focused on software development, but if you have anything along the lines of practical math used in a lab/shop/factory environment around machining, mixing, monitoring, testing, etc., It'd be great to hear about it)

Their fatal issue on the hiring side is that they take too much time to properly score.

If you ask a good candidate to work for 4+ hours, they'll produce code that will cost 2+ hours of a great engineer to thoroughly check.

Lots of companies are very fond of giving candidates huge tasks and having them spend 4+ hours on these. But practically no company will have their best senior engineers (a very scarce resource) dedicate dozens of hours per month to check these tasks.

Hence, these tasks effectively never receive the resources required to properly check them. Especially at the more senior level. If you're hiring a senior engineer, and giving them a 4 hour homework task, a staff-level engineer would have to spend 2-3 hours to properly check it.

Forget about it, that's not happening.

Instead, these teams do ask the senior engineer to take 4 hours to work on the task. However, once the hard part arrives, the team will never check it properly. They'll assign some junior engineer for an hour to check it.

The team won't care, because it's the candidate's time being wasted, while theirs is efficiently preserved. Plus, our junior engineers are so amazing, they can certainly score the work of a senior in less than an hour.

My team is doing our best to create the "perfect" take-home experience and reduce the frequency of these horror stories.

My favorite is a combination: short (1 hr) take-home followed by live discussion/pairing with anyone who does a half-decent job. It reduces stress because candidates will already be familiar with the code (they wrote it!) while being efficient with time.

(I'll note that I have no problem pairing in real-world work scenarios! I love discussing problems with my team. It's the artificial pressure injected by the interview process that really gets to me)

I did a one hour take-home for my current job. I could see someone taking 2-3 hours on it if they were really rusty, and I think that is a good thing because rusty developers can be very good once back up to speed in a language or domain. Live programming wouldn’t give them a chance.

I’ve done multiple different takehomes with some kind of time limit in place to avoid that sort of thing. Tools have this built in.

Also, there are definitely candidates who would prefer a takehome. If you’re worried that a takehome might be unfair, you can offer folks a choice to either do the exercise on their own async, or to do it live pairing with you. You might be surprised by how many will choose the former.

If a candidate chooses to burn a whole week working on it that can be safely left up to them I think. But a take home test is not unfair because they decided to burn that much time on it. It is an attempt to let them give the clearest most accurate signal of their skill level and approach to problems. I would argue that not even pair programming accomplishes that as well as a take home assignment.

1.) You can treat it like pretty much any other engineering job and give a "normal" interview, which will vary a lot by company. You probably rely more on credentials than is sometimes the case with software development.

2.) You treat it more like writing, graphic design, video, photography, etc. and expect to see a portfolio. For fields like that, you might talk about thought processes or how they approach their work but the portfolio isn't really negotiable.

3.) In the case of software, you can maybe have someone effectively create a portfolio for the interview but now you're back to someone putting a lot of time into creating something--albeit possibly something reusable.

I don't like take home tests either, but the point of them is removing the bad candidates from the pipeline so that people who interview them (often ICs, and these are some of the biggest expenses within the company) don't waste time on someone that is unlikely to be qualified.

Personally when I'm interviewing and I get take home tests I choose my battles. I only do them if I really want the job. If it is worth my time.

It's unfortunate, but take home tests are a necessary evil until something better comes along.

I've also interviewed people using both live and a take-home review, and don't have much of a preference there. If the former, I can poke your thoughts in realtime, give hints, etc. If the latter, I get to deep-dive into some of your design decisions.

That's the problem. I think everyone is too focused on technical prowess and forgetting something just as important which is written documentation and explanation.

Pair programming has the downside of selecting people with good interpersonal skills and will tell you nothing about quality of code delivered.

A good middle ground I believe is simple takehome tasks with focus on documentation that could be done in no more than two to four afternoons.

Real world challenges needs time to think, often time to refactor after initial working prototype. A take home allows that.

Also the candidate is already relaxed in the pair programming session because it's a familiar codebase. You'll get signal if they didn't write it or wrote something they didn't quite understand.

Has anyone ever asked you to code for them, instead? So they could learn about you and how you deal with problems? Have they asked how you deal with situations under stress? Or maybe they never did so because they were never comfortable enough to do so, since they were in the spotlight?

I loved giving live code sessions, the more open-ended the better. Give an objective, explain there's no wrong answers, answer all questions, give answers when asked, keep it simple, and gauge response. I found it exposed attitude and knowledge.

How do you feel about take-homes that have an enforced cap at 60-120 minutes to try to remove the competition on time investment?

Do you learn more than it's possible to learn asynchronously? Most take-homes only have candidates write code, but it's possible to understand a candidate more deeply by asking questions about, for example, how they approached the decisions they made.

I agree with you that if the candidate is going to spend the time to do the exercise, the hiring manager should be spending an equal amount of time as well in the process. But that effort can come from the detailed review of the code, the time commitment of which should be in the service of being able to have a meaningful conversation about the candidate's submission in later rounds (which is useful for your last point about discovering what its like to actually work with the candidate).

You're going to learn little to nothing regardless of whether you're face to face or not. Modern interview processes are an exercise in feeding the other side things they want to see/hear. I'm a firm believer that good fit cannot be determined until the candidate has been in the seat for a month or so.

I don’t find it useful to evaluate whether or not a potential hire can map and reduce a list in JavaScript or fix some silly react bug thags so obvious it hurts. Seeing how they approach real problems, architecting solutions on the fly etc are things I enjoy doing but I’m not as much on the hiring side so YMMV

Would be interesting to hear which kinds of tasks you use in a short space of time.

Also, one company had me do a SQL take home which I thought was amusing because then one of the employees nit-picked all the SQL code which I hastily wrote because it was already a risky time investment, even though it was a Python job and I'd likely just use an ORM.

I would argue that a truly qualified interviewer should be able to have a dialogue where they ask you very pointed questions and give you maybe a 20 minute coding problem. If they're not satisfied, they should invite you back for another interview. The major drawback of take-home tests is they are usually very subjective and huge time investments.

I ultimately got 3 job offers, got to the final round at 3 others which had take homes, and accepted one at a FAANG company who didn't use these poor interview tactics but to each their own.

at least with a take-home exercise, i can take my time, do in a few hours, without the pressure of having someone looking at what i'm doing.

How does this sound? I have no idea if this is ideal but so far we didn't hire any more bad candidates than we did using a different process

- There is no reason to take them and this process will slowly fail over time as companies who adopt these policies slowly push away talent options.

- You are not the target audience of a take home assignment hiring process.

I think it's easy to believe the first since it affirms your world view. Speaking as somebody who has a deep pocket of job opportunities and has taken one of these take home assignments, sometimes those opportunities become stale or you're in a tough market or need a job immediately. A number of companies who have bad hiring practices are still fairly acceptable to work at, even if it's not ideal.

TLDR - to pay the bills.

I also think there's a surprising number of edge cases that need to be considered for this challenge. Consider cars that were already parked at the start of the time range, ones that were entered during the range and never left, etc. So I think it does require familiarity with SQL.

Did you disagree with the tagging, the stars, or both?

at least make me _want_ the job before you start placing demands on my time and those of who I might provide as references.

Goodhart's Law will basically apply to any exam under capitalism: "When a measure becomes a target, it ceases to be a good measure."

If these jobs didn't offer more compensation and prestige than other positions, matching candidate competency to job requirements would be much more achievable, but as it is now, it's an adverseal system that demands gamification (because when any candidate plays the game, if you don't, you are actively harming your material conditions with a lower position and smaller salary)

2. The library is organized and the prompts are generally clear and explicit about the requirements to complete.

3. You offer support for candidates with questions about the assignment.

4. Kathy Keating told me about the grading environment you all use with rubrics, blind grading, and rotating cohorts of graders. We actually created similar tooling for the teams we work with.

I also want to explain why they're rated as 3 stars (and why I think this undersells how good they are). When we designed the rating criteria, it was really important to recognize tests that could extract hiring signal without requiring a ton of time from candidates, since that's one of the biggest issues that candidates face. So one of our criteria was "setting clear expectations for candidates (e.g. time expectations)" and another was that the time requested was "reasonable...(<4 hours)". So a test that stated upfront that it would take 8-10 hours would meet only one of those criteria. You can see the full rubric if you hover over the "5-star scale" text in the sub-header.

Unfortunately, the Ad Hoc tests don't specify time anywhere (technically missing both of these criteria) while doing many other things well that aren't captured in our rubric (e.g. candidate chat tool, blind grading). I admit that our rubric isn't perfect and it feels like the Ad Hoc tests are being doubly penalized for something that's easy to fix. In fact, if you add this to the tests, I'd happily update these to 5-stars.

Finally, if you're up for a chat sometime, I'd love to meet you. I really appreciate the work that your team has done to improve the hiring experience for candidates beyond those at Ad Hoc! You can reach me at alex@trytapioca.com

FWIW my team believes that test designers should strive to scope tests to no more than 1-2 hours (ideally 1).

The reason why there aren't any 1-star tests isn't because they don't exist, but because we didn't think anyone would want to see them. Cataloguing these was a ton of work (we sifted through hundreds of tests). Including the 1-star ones seemed like it would only shame companies who used them.

This vicious cycle of wasting time must stop.

One of the ideas my team brainstormed is to create and maintain a library like this that engineering teams can rely on.

How do you figure out if they're top candidates?

First we did screening & technical interviews to arrive at our 5-10 most promising candidates

Right, so we're back at square one doing "technical interviews" again.

And no I don't think you can put any incentives anywhere to get that code onto a Github repo, nevermind a public one.

So now you've put candidates through a silly "live coding challenge" – offering an advantage to the sort of person who is chatty and communicative with strangers in a stressful environment and the expense of people who may be better engineers but slower to warm up.

I really wish folks would be way less dogmatic about this stuff. Every single time I see someone say "take-home exercises are bad", their proposed alternative is also critically flawed.

I have a big folder on my computer of homeworks I've finished for various companies over the years. If I recall I passed all of them, it's sort of annoying that I can't simply hand over this list of other projects I've passed and say "here, all of these companies thought this was good".

What I like about your proposal is that if many other companies did it, the result would be every few years I should take a month of evenings and build out a cool project for myself demoing my ability to write code. If life gets to busy, it's fine if the project ages a bit.

This way you can scratch a personal itch and prep for interviews all the same time.