Anyone else share in this frustrating experience? Have you successfully asked for feedback before?
As far as I remember, I was told it would take a few hours, maybe 4, but the assignment looked rather fun, so I thought, whatever, let’s do it.
Maybe I didn't understand the assignment fully or missed some cues; I don't know. But it took me roughly 10 hours spread across two days. After 5-6 hours, you don't feel like throwing the work away; you just want to finish. It was pretty frustrating.
After returning the assignment, I waited the two weeks they asked for and heard nothing. I sent one email, waited a week, got no reply, sent another, waited some more, and still received no reply. I ended up sending a handful of emails to various addresses I could find. I even sent a DM on Twitter to one of the founders to let them know. No reply anywhere.
Overall, it was a pretty bad experience.
The job listing promised mountains and that it will take few hours and they’ll take time in their feedback,etc.
except i just got a generic reply that “we’re not satisfied with the solution”. For something that took ~10 hours, i was at the least expecting a vague pointers on whys. I even sent an one pager explaining style decisions, caveats, etc. It was pretty insulting.
And when asked for a follow up, got a more generic bs about how the evaluation criteria is honed from their years of experience and that is not something they share outside.
One good thing that is it made me realise interviewing still sucks and i just stopped looking for jobs.
Was it my issue waiting a short time to start it? Maybe, but you’d think they’d at least try to answer a question or two. Oh well, I just gave up because if they weren’t going to put in any time, why should I?
Don't get me wrong if you had fun (and I enjoy them too) nothing wrong with that.
I'll make sure we get you a response within the next week.
Feedback was something like "Assignment is private and you can't store code on GitHub, you should send zip archive"
The most fun task I had so far, the feedback was pretty good as well. Kudos to tptacek for that.
Maybe they had bad experience with sharing feedback directly?
If they said it would take 4 hours and you spent 10 hours on it, it’s likely your approach was ineffective compared to a more skilled engineer.
I’ve passed interviews at demanding companies like hedge funds, Google, Facebook, Snap. Not once have I “passed” a take home interview even when I know I got the solution correct. Sorry, but I refuse to do them now. Companies need to be able to show a token investment in time with an actual engineer interviewing me for me to be convinced they are actually considering me as a candidate and didn’t just send me a takehome link without any due diligence because it costs them nothing to do so.
A nice way to say this is, "I'd be happy to complete this takehome assessment, but it will take a significant amount of time and first I would like a short phone call to ask some questions and get a feel for the company, nothing too long, I just want to make sure we're all on the same page before I spend my time on this. When can we schedule a call?"
I wish people were less surprised when we tell them they deserve to be paid for their work, but companies have been advantaged in the hiring process for so long (excepting COVID) that disregard for people’s time is embedded in corporate culture.
However, to the point people have made about not getting comprehensive feedback: it’s just not worth it. Corporate counsel will basically refuse to let you tell a candidate why they were rejected for fear of any inadvertent lawsuit exposure. This is also deeply embedded in corporate culture and it’s hard to imagine it changing.
Twice I've aced these challenge with time to spare and was then rejected or ghosted.
It's the same idea; don't invest time into a company's interview loop unless you know they're investing something too.
I had this experience a few weeks ago with Wolverine Trading.
It seemed clear that they didn't respect my time. Perhaps I dodged a bullet.
I have had the exact same experience. They have always been a complete waste of time for me and I have since adopted a policy that any takehome challenges = automatic close the loop for me. It's not worth explaining why either. If this is your process and you find it appropriate to offer it as a first option, then it's just not a good fit.
Not joking. You hated interviews before? Get ready to rock.
Long story short, the company passed on me, saying that my code was easy to understand and high quality, but I didn’t handle all of the corner cases. I listed most of the corner cases they mentioned in the readme as limitations from the time limit. It was clear that while they said to limit my time on the project, they didn’t really mean it. They wanted an exhaustively finished product in a ridiculous amount of time.
At this point in my career, I’ll just decline interviews that don’t respect my time.
I obviously noped out, but am still wondering if they are aware that all the people working there 100% cheated on their take homes.
I wish I had saved a copy of the assignment. It was ridiculous.
The worst part was that after working there I found them to be terribly unproductive and it would have likely taken a team there a week to build what they were suggesting...
But yeah as you said, either ignore the time constraints if you really want the role or just politely decline due to the red flag / inconsiderate nature.
And I'm like.. an hour is not long enough to implement any reasonable webapp... I have no idea if other people went over or not, and it almost feels like a test of commitment (if I was serious I would take the actual 4 hours it would take and pretend I did it in 1 or something like that).
1. Other candidates were clearly spending 8-9 hours on the problem. I saw their commit logs. A commit every 1-2 hours for 9 hours. Several candidates public repos were like this. The assignment was like 2 pages of requirements supposed to be done in I believe 1 hour.
2. They told me (thru the recruiting firm; I never spoke with them) that it would have been resourceful to use the other candidates solutions.
> ... as I wasn’t paid for the interview and I didn’t feel it was fair to me to spend any more time on it than that.
I have been paid for an interview, a coding interview where I fixed a bug and added a feature while my pair (future colleague!) chilled .. I think he surfed and I'd ask him clarifying questions occasionally? Anyhow, I got paid like $400. Best interview ever.
This is my philosophy now. In my younger days when I had no experience I had a lot of my time wasted by recruiters and companies - never again. Now I ask up front and politely what the interview process will be like. If there are any home projects or mentions of whiteboarding I end up passing. No big deal - I know there are probably loads of developers willing to waste their time competing for the position, but not me.
It's worked very well for me and eliminated a lot of stress when I've been looking for other positions. The only negative I can see is the strategy wouldn't work great when in between positions and really needing something, which luckily I've never had to experience. I suppose in that case I would force myself to go through with the hazing process.
We don't fail take homes because of edge cases though. There are things we expect people to handle in the challenges that are important, but I don't think more/less time really influences those.
Honestly 2h from scratch is a rush to do anything? However simple it is, you're asking for an 'entire' greenfield project, of course I have skipped things and missed corner cases.
Why is that? Personally, even though I spent two days working on it and ultimately failing it, it was one of my favorite interview processes ever... way more efficient use of time for both parties, IMHO, and a better way to measure the actual skill set. And it was fast! I did the assignment in a couple days and then heard back from them in another day. That's way better than the other interview gauntlets I've had to run, which usually included 3-4 rounds of talking to a bunch of people, with weeks or sometimes months in between, and most of those folks are people I will never see again after the interview cycle.
And there's also not the absurd pressure and ambiguities of real-time whiteboarding or live coding, which is not at all representative of how people normally work (with access to the internet, their preferred workstation and IDE, etc.). Having gone through one of those sessions, THAT feels way more like a "dance, monkey, dance" bullet dodged vs the casual take-home.
By contrast, a take-home gives me a "just pretend it's a normal work day and show us how you work" situation and I can focus on the problem at hand instead of the politics of hiring. Isn't that a good thing...? What am I missing?
[1]: My complaint was about the lack of feedback after a take-home, not the take-home itself (which was actually illuminating and fun): https://news.ycombinator.com/item?id=36446949
I'm not saying that all interviewers are lazy or bad, but most of the ones I've worked with are. Hell, maybe even I am. Time spent interviewing is time not spent working on things that you might actually be incentivized to do well.
I’d rather spend a day preparing a D&D game for 6 people who will have fun for 4 hours. Once the game is over, we’ll have memories of a time we spent together.
Alternatively, I’ve spent a lot of time playing with Home Assistant. The end result has been automation I use on a daily basis.
That’s why I don’t like them.
A "takehome and debrief" call wouldn't be too bad as an alternative to a full day of panels, but I've never personally come across it.
I don’t have two days to give away for free to potentially multiple prospective employers. Why? That’s none of anybody’s business.
How do you qualify “build us a full system with tests and everything in 6hr plx”?
Speaking as someone who absolutely hates live-coding something during an interview (unless, maybe, by chance it's a practical exercise ... something like "build up this simple CRUD web service" ... not "solve this leetcode problem" ... but this is pretty uncommon in my experience, unfortunately), I rather like take-home assignments. As long as they don't take any more than 1-3 hours. I've been in situations where I've declined a take-home assignment that was given to me where I estimated it was going to take 10-20+ hours. No idea why any company would think that is reasonable.
For myself, what I hate is when the take-home assignment comes first. Like, before you talk with anyone at all, or maybe immediately after you did the 15 minute HR/recruiter screen. If I get a take-home assignment at that point, I decline.
There's no denying the fact that a take-home assignment is a large investment by the candidate on this random chance to get the job. I'm not willing to make that investment unless I've gotten a chance to talk to at least _some_ of the people I'd be working with at a company to better gauge what the situation is and whether I think it's a good fit for me. Even a 15-minute HR/recruiter screen is not going to do that. So yeah, if they just throw the take-home assignment at me right off the bat ... yeah, no thanks.
This all being said, yes, it is frustrating though to have to go through a bunch of these and get rejected with no feedback, or just ghosted, yeah.
If you ask the majority of job seekers here, the only correct way to hire is to hand out 300K/yr offers after a casual 30 minute chat.
I didn't end up getting the job but it was easily the best and least anxiety inducing interview I've ever been through.
Well, that's the issue. It may be because I'm a bad programmer, but I can't think of a take home that took me less than 3 hours. The take home I did for my first job took some 20 hours over 4 days, and I exaggerated and said it took 12 (which didn't garner a response so I'm guessing that wasn't an unusual answer). I could do that while I'm a student, I can't do that again as a working professional without wasting an entire weekend after a week of full time work.
I still hate leetcode more, but at least there you have an explicit timer.
>For myself, what I hate is when the take-home assignment comes first. Like, before you talk with anyone at all, or maybe immediately after you did the 15 minute HR/recruiter screen.
I've never had a test come later. 15 minute interview call to make sure the high level details are correct (pay, location, physical office vs. Remote, etc), and then I am sent an interview test.
Granted, my last 2 jobs did not employ take home tests, so I know I'm not forced to do them. But given my experiences I understand why others would be opposed to them. It sounds like you would be opposed to my experiences as well.
Oh, but onto your questions. So yes it is frustrating. And as far as feedback. I stopped asking for feedback because it was always maligned. Nobody will really know why you didn't get hired, remember, these companies aren't really hiring unless you are young and cheap. In most cases my feedback sounded like it was almost for someone else. Like "needed more linux experience (I haven't touched a windows or mac OS since 2005). Or my favorite, after talking about a data warehouse I built to house 5TB of data and 15 billion rows, and all the different schemas I migrated through, their reason was they wanted someone with "more database experience".
Nah, I think there are more factors at work (Also, cheap is relative, therefore dependent on other factors, and therefore somewhat vague)-- its a multivariate situation where you, and the many factors you bring, flow into a process and the many factors it brings. If enough of those factors line up enough-- you get hired.
If it takes 18 months to get hired, I'd analyze the factors you bring. Such as: Web portfolio & Online presence. Network. Resume posted on multiple sites. Appearance & Demeanor during interviews. Skills/Years Exp/Qualifications relative to the role's stated requirements. Etc.
I landed my first job at BigTech at 46.
I changed jobs 6x between the time I was 34 until I was 46.
I doubt that I’ve done more than 40 interviews over 25 years between 8 jobs.
> these companies aren't really hiring unless you are young and cheap
Got rejected a week after submission with no explanation. It was a really good problem and I had fun solving it and it satisfied all the requirements correctly. When, disappointed, I asked them for feedback, the recruiter simply said that they "felt that it fell under their expected level" and that "The team has a very high bar set for this role and have made it a point to make sure people who join the team can contribute immediately which unfortunately means tough decisions like this have to be made". They didn't share any specific feedback on my code despite my following up for more specifics. Really pissed me off.
As a wise man said, it is possible to make no mistakes and still lose. That is not a weakness, that is life.
You probably dodged a stressful position. Companies or teams that can’t allocate time to onboard talent are likely poorly run and can’t afford to invest in people. “Contributing immediately” is a fair indicator of that problem.
Believe it or not, I've also gotten takehomes which took multiple hours, involved some non-trivial data retrieval and management, and gotten rejected with less effort than it took them to prepare the ask.
Later I had a sinking suspicion that it was not a real position, what it was was three brogrammers who found a way to get unemployed rubes to write code for them, for free, so they could deliver it themselves without having the requisite talent.
Are you positive that didn't happen to you too?
There's a perspective starting to come up, ie ChatGPT is ruining take-homes. This tweet[1] is one example from the hiring side, but I wouldn't be surprised if it ends up flooding their inboxes and your submission suddenly is competing for time with a bunch of bots.
[1] https://twitter.com/djsmith42/status/1661156114883567616
Well, you do have a choice. You could tell them that they were rejected for copy/pasting or because the requirements were not met. You don't have to do into more detail than that.
The report can be 1-2 sentences. It doesn't need to be meticulous or a "boot camp".
If-so, you're doing yourself a favor by providing feedback so that they might improve by the next time they interview.
A friend and I tried to start a company around this, basically resolving three big issues we saw with take-homes:
1) No feedback. If you spend a couple hours on something you should get meaningful feedback.
2) No clear criteria. Is the hiring company going to slam you on correct syntax? If so, you should know that upfront!
3) Time guidelines that had no enforcement and thus led to an arms race where candidates would spend ever-increasing amounts of time and skew the standard of what "good" is.
So we fixed those things:
1) We provided the feedback, not the hiring companies, so legal liability was non-existent for the hiring companies. We double-blinded the process as much as we could (evaluators didn't know who the candidate was and vice-versa).
2) We told candidates upfront what they'd be evaluated on. Not down to the level of "you must implement this problem using a max heap", but we would say something along the lines of "The company is looking for an academic algorithmic solution to this problem" or similar. We would then only allow evaluators to evaluate them on these axes and nothing else.
3) We also strictly enforced time limits by basically telling candidates "hey you'll have 2 hours to submit from the time you hit start and see the prompt, so please make sure you have two hours from when you hit start." -- not ideal, obviously, but the best we could come up with to resolve #3 above.
As you can probably imagine, the market just wasn't really there for this. I think candidates generally enjoyed it in comparison to the vague, unending slog that most take-homes are but:
1) The value prop just wasn't really there for most companies: They mostly use these types of evaluations on more junior candidates, and unfortunately the hiring market for junior candidates is highly skewed towards the employer.
2) More surprisingly, we realized the time their current engineers and managers spent evaluating these takehomes just wasn't really a consideration for them. We tried to frame it in terms of "here's how much it costs you to evaluate these take-homes wrt time spent vs. us", but it was a difficult sell regardless.
We actually had the most success evaluating candidates from more non-traditional backgrounds upfront ourselves and then charging a placement fee if they were hired, but we ultimately didn't really want to continue that.
employers don’t know, or seem to care, how much it costs them to replace and/or onboard somebody. usually because they have no idea how they’re off/on boarding hires.
not too surprising that they wouldn’t care about the cost of one component of the hiring process.
do they just want to see that yoiu know how to load a csv file and make a barplot? or do they want a showcase of advance statistical modeling to see where your ceiling is?
If you are interviewing as a software engineer, you should be able to write FizzBuzz. Yes, it is stressful. Yes, I hate leetcode too. But, a company has every right to check that you at least know the basic skill you say you do.
If more engineers start to refuse these kinds of "dance, monkey, dance" processes, they'll start phasing them out.
Then the hiring manager came back, "all looks good! I need you to interview with the team, the CEO, the other team" and what not.
I declined and said that I'm not spending 10 hrs in a process you asked me for (as in I didn't apply, they reached out to me). I got an angry email saying "what do you mean? We have only met with you for two hours?". I pointed out that it takes time to do the tests, then nothing more than crickets from the company. Not even a thanks for your time.
If you and your team cannot suss out someone's capabilities from their CV, portfolio, interview, and references then YOU SHOULD NOT INVOLVED IN THE HIRING PROCESS.
It's a shit-test that you believe helps filter out under-qualified/unmotivated candidates, but it actually has the opposite effect:
Experienced candidates (you know, the ones you actually want at your company) just move on to lower-friction interviews where they don't have to start jumping through nonsensical one-way value exchanges on Day 0.
Leetcode grinding for a whiteboard is a worthless skill outside of the interview. I'd rather code up something. If the project is over burdensome you can always pass.
The best homework I got was a "working" project that purposefully had bad practices scattered throughout. You could elect to fix the mistakes or just document them for the interview.
The company doesn't want to spend time getting into an argument with the person over it or potential legal issues arising from such feedback.
Companies refuse to even read a resume/cv these days. They make you sign up for an account specifically with their company and jump through all their hoops and type into fields exactly the information covered in the resume/cv/cover letter. They have you write out answers to questions they later ask you again in person.
Of companies don't want to spend the time that's fine, but they shouldn't expect prospective employees to waste exponentially more time. And arguably the job hunter is in the more desperate position, as evidenced by the fact we are willing to jump through all of the aforementioned hoops. Unpaid. For a chance at something. For which you may receive no feedback.
Fuck that it should be illegal to do so unless they compensate us.
I don’t want to either, but if the feedback they send is clearly bonkers then what am I supposed to do?
If you have a dialogue in the first place you can correct their misconceptions immediately, but if they use take home the only time you can do this is after the fact.
Got a "Thanks, you're not moving on" message and that was it. Man, the rejection is totally fine, but I really do wish they could've spent a few min just explaining some details. I've gotten more thoughtful rejections from just cover letters and resumes, not to mention public PRs in strangers' repos.
After spending so much time, it just feels like a betrayal of some unspoken developer ethos (vs say talking to a generic HR screener). If you're going to make someone code for you for hours and then dismiss their work, please at least tell them why in just a few sentences. You don't need to comment line by line in a code review, just general thoughts like "better code org" or "poor architecture and readability" or "better tests would've caught this major bug" or whatever.
In my case this was a small company I was super excited to work for, and waited more than a year to finally have a chance to apply for (once my current job ended). It was definitely disheartening and makes it hard to want to try again with them in the future.
But, you know, the other side of the coin is that maybe they're just getting swamped with so many applications they can't take the time to thoughtfully answer each one. I imagine I'm competing against a horde of more qualified ex FAANGers right now and maybe they're too busy trying to decide between the top 3 or 4 vs the long tail of hundreds of us who failed the take home. Who knows...
Maybe. At the point in the process where you're seeing someone's project, you've already spent time probably phone screening them, sending them the project, hopefully reviewing it. It takes relatively little time to write a quick sentence or two in review of the assessment. It would be nice even if they didn't look at the project and just said "sorry, we didn't have time to look at your project because we're moving forward with other candidates".
The responses are arbitrary, and mostly based on the reviewers preferences.
And then knowing that what they think you did wrong makes no sense, you still have no recourse.
In the past, when I was a middle-of-the-totem-pole decision maker, we administered a take home. I wasn't really empowered to give feedback, but as far as I know nobody did. The best take home test had a joke in the comments. We were a game company, so it made sense for us - you have to care about entertainment.
Today, as the ultimate decision maker, I ask only to see pre-existing code with git blame turned on, and I review it on screen with the candidate. In this context the candidate receives feedback, and it's almost entirely focused on style because IMO that is the single greatest predictor of actual coding experience.
Let's discuss the underlying issue here: ineligibility, of like 30-50% of candidates. In this context, take homes make sense, because they occur in the absence of any other legitimate reason you should talk to someone.
For example, here are my tech company LinkedIn Talent Solutions screening questions:
Have you completed the following level of education: Bachelor's Degree? What is your level of proficiency in English? During our call, will you be able to share recent source code, wholly authored by you, of a game, application or website you worked on?
Out of 20 candidates, 12 could answer Yes to all three. What exactly should we do with the remaining 8? I can see how a take home can address the ineligibility.
If after 3-4 rounds of systems interviewing, the company came back and said: "Hey we like you, and we're interested in making an offer, but we want to check your skills..." Yeah, I'd do the assessment. Especially if I did 1 round of 45m coding. Without that, the company can't verify that I have "FizzBuzz" level skills, and that I didn't cheat on the assessment.
When it is upfront, it is rarely worth it on either side. Companies lose a large fraction of the engineers applying into their funnel, and honestly as an engineer, it is rarely worth my time. I can spend that time interviewing at other companies that respect my time.
As with many things in life: It is when, and how things are done. Not what things are done that often matters.
Even just a "hey, we're genuinely interested and you made it to the shortlist" or something to that effect would just make a world of difference. Now I have no idea if I'm on any sort of shortlist or if they're just shotgunning this to everyone. I tried asking a few times, but I always got such vagueries in return that I might as well not have asked, so I stopped bothering.
Would also help if people made the tests at least a little bit interesting. Now it's either "do this boring CRUD thing you've done 1000 times already" or "solve several NP-hard problems that have been subject of Turing awards within an hour".
The last time I spent quite a bit of time on one of these things because it was interesting, and even put the result on GitHub (with no mention that it was for a take-home test, it "just" looks like any other project, also slightly modified to fit my needs). Then again, what's "interesting" for one person is "boring" for another.
Coding exercises I’d be ok with:
- fix all the bugs in this repo to get the tests to pass
- implement a feature to show basic understanding of a concept
- do some quiz or puzzle to show you can achieve a discreet objective
- refactor some code to clean up a mess
- show you can use their SDK in a sane way
Ways to evaluate a coding exercise I can support:
- if you can complete this exercise, that’s all we care about
- we’ll use this exercise to implement another feature in a pair programming exercise
- using it as an introduction to the type of work one might be doing
- communicating that we like to give candidates a chance to code without the pressure of someone staring at them in a live interview
- promising an up or down vote on every exercise submitted
- doesn’t require back and forth with a someone to ask a bunch of questions, mostly because it really drags out the interview process
Basically if a take home exercise anticipates all the common objections to a take home exercise, tries to address those expectations, and has an attitude of “this sucks for us as much as it sucks for you”, I’m happy to jump through reasonable hoops.
One of my take homes had a follow up like this and I did enjoy it. The interview went well but then I got rejected with very confusing feedback. It's hard to win as an interviewee.
Every candidate that we are interested in from the resume pool gets a screening call from an engineer. If they pass it, they get the option of their technical test being take home or over interview.
If they do the take home version, we have a marking rubric for the features that we are looking for in a solution along with sample solutions at the different levels. Anyone who does the take home assessment gets feedback about their submission based on that rubric so that the feedback is consistent across the multiple interviewers.
In some cases, we also share our solution if the candidate had follow up questions on the feedback that was provided.
...but both paid me a great amount of money to complete them so I’m not complaining at all.
Paid take-homes are definitely my favorite way to interview. The process doesn’t scale past 100 or so hires but if your company hasn’t hit that point yet I recommend it.
But in the end, they rejected my project though with no feedback. Can't win.
With the rise in number of new security engineers all competing for few "security research" jobs (security research/hacking is the "I want to be a game developer" of security), you start getting into these convoluted hiring processes. Unlike standard software engineering, there aren't even remotely enough positions to accommodate everyone, so the bar can get absurdly high.
Honestly, if the team is asking CTF questions, they clearly want hires with previous CTF experience and should just do targeted hiring from the top teams at different conferences.
At least send people a free t-shirt if they complete the challenge.
Please cite the relevant court cases.
The irony of these claims is that the possibility of lawsuits doesn't actually stop companies from doing terrible, lawsuit-worthy things. They do such things all the time.
Occam's razor suggests the explanation is simply that companies don't care about job applicants. And this is demonstrated in many ways.
Not to "1-up" you, but I once interviewed at a company where the interviewers were not only not allowed to tell me what I got right or wrong of their questions (where there was a right/wrong answer), but were strongly discouraged from any sort of non-verbal cues about it.
I was told later this was to prevent interviewer "A" from giving me any answers that "B" might later ask. Insanity.
Someone much later said this was their "Kobyashi Maru" interview. I didn't pass it; probably for the best.
Yes.
At this point I view the practice as a signal the position wouldn't be a good fit for either party.
As are interviews of any kind without feedback. I'm not referring to holistic feedback (let alone an explanation for why they don't move forward). I mean the kind where they grill you with question after question (often poorly phrased and sometimes insultingly basic) and don't provide any indication at all as to whether your answers are good or bad or not.
They just keep going on, poker-faced - saying, in effect: "Don't speak unless spoken to".
I mean you'd be surprised at the number of people that can't write a for-loop ...
I have no idea why anybody invests hours of their time in these situations without some guaranteed return on it.
Why do programmers accept to spend all this time on a process that might end up in a rejection? I don't mean just on take home assignments, but on the grueling in-person process that can take days or weeks as well. It's a collosal time sink with an unknown outcome.
The interviewers are paid for their time spent interviewing, so it's only fair that companies respect the time of the interviewee, and pay them as well, whether they pass or not.
We often forget that interviews are a two-way street. The person is interviewing the company, as much as they're interviewing them. The risk if it doesn't work out should be taken equally by both sides.
If a company doesn't respect your time, it's probably a signal that they will disrespect you in other ways as well once you start working there, so it's best to move on early.
It's a nice dream, but I really don't think it's feasible outside extremely niche scenarios. And those can be (and often are) quite a bit more flexible in how things are run already.
Both companies and candidates have realised that the signal:noise ratio is against them.
Companies solve for this (badly) by cranking up the funnel, but then are poorly resourced in handling that many suitable and unsuitable applicants.
Candidates solve for it with recruiters, keyword-packed CVs etc.
If we’re honest, hiring just sucks at the moment.
Perhaps we need a better way of ensuring job specs are correctly self-discriminating between good enough and not-yet—and can drive useful feedback for all, and a better way of handling that first pass that ATS-screening so that take-homes can serve a deeper purpose and be resourced correctly.
“To be honest, a lot of the work is refactoring 20 year old technical debt. It’s tedious work for anyone that doesn’t enjoy that, but the work pace of the work is reasonable & we understand that it takes time to confidently make changes to our barely tested spaghetti code. If that sounds like your kind of work, we want to talk to you!”
One of my coworkers at Amazon was leaving for greener pastures. Before she left, for the lols she showed me the feedback she left when she interviewed me. It was something absolutely brutal along the lines of "doesn't appear to be able to code at all" (I was offended upon seeing this "feedback"!). My point being: interviewing is stupid, and, generally speaking, there is no valuable feedback to be had from the process.
But really, I think that interviewers should be gracious about takehomes. And if they're not, that gives you and indication of what kind of place it is to be.
Adjust your sights on those and it's lovely. More often than not those companies are great to work for across all metrics.
Most of these leetcode interview companies just follow what Google does for that reason alone. And Google does it because they have to cull 20,000 applicants somehow in the first wave. They just _can't_ interview that many people from the jump.
Platforms like Woven do this automatically, but they miss the mark. A good half (if not more) of the value is observing the candidate working through the problem and responding to collaborative feedback, not in actually solving the technical issue in question.
"Why should we give you feedback and basically prep and improve you for your next interview at a different company?" If you are given feedback, the company is basically doing the prep work for the next company at their own expense. There is no ROI in giving you feedback if you did not make it.
It's harsh, but true.
The idea is, if you submit to company A, they rejected without any feedback, let's other companies know about it, and both of you is in a win-win position.
I recently completed a take home challenge for a full-stack engineer role. Given a week and told I could request more time if necessary. Instructions specifically said "We're not looking for a pretty UI, we prioritize component structure and functionality." and "Use any framework you'd like." The challenge was to build a back end with four endpoints for CRUD operations on a resource representing an application for car insurance. The front end was a single form for updating and submitting the application.
So naturally I whipped up a solution in less than 24 hours with simple pre-built components that both looked great and were functional (Mantine UI + Firebase)
I submit the challenge and check the logs every day to see if they'd run the application. A week passes before I hear anything back from them, logs still showing that the front end was never visited, none of the CRUD endpoints ever submitted to. "We reviewed your submission, thank you for your time, best of luck in your search"
Naturally, I respond confused about the claim to have reviewed my submission, wondering how they managed to test the functionality without visiting the site or making any submissions. I asked if I had misunderstood the challenge, asked if there was any feedback about how my submission fell short of their expectations.
"We reviewed your submission based on the code you submitted and came to the conclusion not to move forward."
...yea long story short, I'm never spending time or money on a take-home challenge ever again.
I'm on the other side and having to reject dozens and dozens of candidates after they spent 30+ minutes on these takehomes, and can't tell them why.
My only solace is that 30mins on a takehome beats an hour in a live interview. (my other solace is that we try to make the questions interesting and non-trivial, and we encourage candidates to use ChatGPT and then edit the answers, since the questions are just a bit too tricky for LLMs)
(hiring DevOps/SRE for wide area distributed compute: https://www.joinmassive.com/jobs?gh_jid=4236238005 )
It turns out that that position was eliminated within 12 months. If I'd been hired then I would be unemployed by now. So I'm amazed and grateful for my current position, with an incredible flexible schedule, 100% WFH, and all the rest. Sometimes things work out for the best when we least expect it.
I bet the hiring company got a bunch of "fine" or "ok" project submissions from candidates and then made somewhat arbitrary choices on who they would move forward with.
If I'm presented with one, I take it as an immediate sign that I'd be a poor fit there and move on.
No feedback at all, or even any aknowledgement that they'd looked at it, until a couple of weeks later with a request for a tech inteview.
I just refuse to do others, sorry. If they want a hands-on test, they can pay me to fix a ticket, or even just pay me to do a test project. It's a small sum for them - for me it's multiple hours, over multiple interviews, all for free, it's basically a full-time unpaid job, just BS.
Oh, and the project smelled like something they actually needed in prod.
A couple of years ago a company send me a detailed take-home project that was supposed to take about a week with an engineer assigned to answer any questions I had, and by the day before it was due, neither the engineer or the recruiter hadn't bothered to respond to a single one of my three questions I replied with over the next few days, including a basic yes-or-no clarification about whether one of the libraries they listed as okay to use included one of its optional but extremely common extensions. When I replied on final time on the thread saying that spending any more time on the project would be a waste of my time and that I was no longer interested, the recruiter finally responded a day later saying "sorry, that engineer was busy" and adding a new engineer to the thread.
It was a waste of time for me, but I also realized that I probably dodged a massive bullet by not working there. An engineer not proactively telling the recruiter they won't have time due to unexpected work or a recruiter not noticing the lack of response from the engineer assigned to their candidate might be an individual failure, but both happening without anyone involved seeming to notice or care is a pretty good sign of dysfunction at the organizational level.
1. 30m phone screen with the hiring manager with standard interview questions and explain the process.
2a. Ask candidate for code example. If they have OSS work or personal projects we can look at that first instead of take home. Very few people take this path which is surprising.
2b. Take home GitHub exercise. Fork a repo for the person and have them read the readme and open a pull request.
3. 15m code review from hiring manager. Filter out very poor communicators or people that don’t know the language or know it at the right level.
4. Dole out PR review to a lead engineer. PR review for day to day work is extremely high priority. Fitting in a take home usually can get picked up in a day too.
Easier said than done. I’m sure all candidates that are passed on are frustrated and would like more feedback. Sometimes they ask for followups with more but I push back if I think what was given is fair already.
The hardest part to me as a hiring manager is that hiring is a winner-takes-all. If someone has a good submission but there are two more people in the pipeline what do I do?
It’s also hard to do everything perfectly and timely for a candidate given all the other priorities going on.
I hope some candidates take comfort in how the process reflects the company. If you’re unhappy with the process you get you might be unhappy working with those people.
But if you’re unemployed you might be happier employed than not…
This also means something about the person running the test, though. They are not experienced enough in the engineering hiring pipe to know what to do (i.e. they are unable to differentiate good engineers from bad engineers) and either do not know or do not care about the adverse selection. This means their team is likely to be mostly mediocre engineers. You should pick it only if you need a job.
Essentially, the interview process is part of how you find out whether the team you're going to be working with is good. If you believe a certain process would not result in being able to differentiate a high-quality eng from a low-quality eng, and they're running that process: you should increase your likelihood that that firm contains many low-quality eng.
My biggest learning about software interviewing is spotting immaturity. I have it down to three points:
1) The only purpose to software is automation. It’s not data, entertainment, or anything else. If an internal team is not aggressively applying automation like static analysis or automated tests to their code they are not mature and I will not work there.
2) Does the team or employer understand the software platform they are hiring for? Software platforms are things like web browser, a game engine, terminal emulator, and so forth. I most commonly see over confidence about this when the employer is fully reliant on some framework to do everything. That is serious code smell because any deviation or problem outside the framework results in serious emotional problems. I do tire of emotional junior developers trying to define my job as some sort of infantile pacifier.
3) Can the team communicate in writing? I mean specifically the other developers.
These failures are what I see in 95%+ of software employers. I am willing to take a massive pay reduction to join an employer that doesn’t fail from such immaturity.
For example at one place here, Centro Ático of Pontificia Universidad Javeriana, they assigned people to create a whole website (several pages pulling media) with plain HTML+CSS+Javascript, which I did. I don't remember how much time they gave but it took days. Even I pulled an all nighter on that because it was that demanding and I had a full time job at that moment.
After that, they selected the ones they liked the most and called them for a presential interview. They called me, I had to make up an excuse at work so I could go to that.
Ghosted. Never heard from them afterwards.
Same with many other places where they make you feel at home and the communication seems to be great - most recently, at "RebelMouse". With the Silicon Valley Bank thing they told it wasn't a sure thing they would hire someone. I did the test anyway. After several weeks they told me they finally were reviewing my test. But more weeks passed and never heard anything. Asked them about that, even saying something like "I don't think you're the kind of company that ghosts people!"... But turns out they are.
The worst though is that some companies are allegedly (cough, Clipboard Health, cough) using job applications for free labor according to Glassdoor reviews.
www.geektastic.com
We provide take-home assessments as a service for hiring teams. We have a team who carry out reviews on candidates' code challenge submissions in their spare time (paid of course). Their focus is to provide detail to the candidate that the hiring team hasn't got time to do. Our bar to join the review team is set very high and we are assessing their ability to provide well balanced feedback.
Every submission gets a detailed line by line review, with detailed explanation from our review team explaining the pros and cons of the submission. We're looking at things like code quality, solution design, problem solving skills and test coverage.
We also provide the ability for the candidate to write comments on the review to try and avoid the usual one way street that happens with lots of technical assessments. I have seen these comments get a candidate a next stage interview when their initial submission was deemed not to be a 'pass'
Each challenge has a set of review guidelines that aligns of team of reviewers to standardise reviews.
Recruiting is one of those things that is beyond broken and no one knows or cares how to fix.
I interviewed for all types of companies, big and small, from Google to Amazon to small startups and local university spin offs.
Overall the experience sucked, before, during and after the interviewing process.
I remember at Amazon, we (candidates) being welcomed by HR staff wearing shorts at a business center in Philadelphia.
Although this may not look like a big deal and in the spirit of a tech company, it really tells you about the judgment some of these people have and those who hire them.
If they don't care about throwing on some minimal business-casual clothes to meet up with grown-up professionals who took the time to flight from all over the country to be there, how will you expect they will give a damn about anything else during the process?
This is an extreme case for sure but I have noticed the same unprofessionalism at all levels, not only dressing codes, anything from being late for the interview, to last second changes poorly communicated to the candidates, interviewers looking at their phones or laptops while you are talking to them to even missing appointments altogether.
I did have some good experiences, one small startup out of MIT labs aced the process so well that I borrowed some of their methods for interviewing myself at my current company.
But in general, the recruiting/hiring process is unprofessional everywhere and I'm just glad I probably will never have to go through that manure again.
A (reputable) company asked for a take home assignment after a "group interview" as a college hire, and as a college hire I didn't know to tell them to pound sand.
Same experience as yours, I spent twice as long as I was told it would take, then was rejected with feedback that was generic and very subjective.
Admittedly my career went in a different direction (company was realtime ads, I went down the distributed systems rabbit hole) but I still disagree with their feedback. Given the chance to do the assignment again (assuming, for the sake of argument, that I wouldn't laugh in their face) I would come up with a very different solution, but still one that I think would yield similar wishy-washy but negative feedback.
Moral of this story is to never accept take home assignments, and don't get discouraged if you get poor feedback from one because it's essentially a distilled version of everything shitty about onboarding & code reviews, rolled up into a single package
No one owes you anything. If these companies display poor behavior during the interview process, add them to a list, name and shame, whatever you think is appropriate and move on.
Tbh if I didn't have knowledge on the framework I know I wouldn't have been able to complete it. However I found it disheartening the fact that they just checked that the app worked, didn't have any bugs but didn't even cared to ask around if I actually did it.
These days I can't even imagine how easy it is to cheat with ChatGPT. Last guys I have interviewed I tell them to download a repo with a sample app and poke around with questions. I can quickly tell if someone is BSing and how deep they actually know about the language/framework. I've found this to be better than live coding or take home.
On the other topic, remember that the hiring side doesn't have an incentive to tell you how well or how much you suck at coding, interviewing etc.
How did we do it? Limited time, feedback, and benefit of the doubt.
We put a time limit on the exercise. It was a 1.5-2 hour exercise, with a 3 hour cap. Some felt this added some stress, but many appreciated that it didn't take up their whole weekend or multiple evenings. In the few instances that candidates overran this massively (~8 hours) without notifying us in advance, we saw no difference in their scores.
As for feedback, we were scoring against a rubric that was quite detailed across multiple different categories. This forced us as reviewers to break down and justify, not just say "I don't like it". And because we ended up with 5-10 bullet points of things we did or didn't like, there was useful feedback we could give. There was also the added benefit that we all knew the problem inside and out, and so could drill down to the essence of the solution quite easily.
And lastly, the benefit of the doubt. No software engineer writes perfect code every time, and code review is an important part of the job, so if the feedback boiled down to "I don't like it but it's close", or if it was something that with regular code review would have managed to pass the bar, then that's what we'd do. We'd schedule another round to discuss the code with the candidate, see how they responded to the feedback, and hopefully move them on to the next round.
Why doesn't everywhere do this? It's actually pretty hard. You need a recruiting team who are very on top of the logistics and running the process, and you need reviewers who know the problem well, are bought in to giving a great candidate experience, and who are good enough to be able to judge others. In my experience most places don't have much of this.
Take-homes are much better than Leetcode interviews, but smashing yourself with a hammer in the thumb is much better than in the face.
I think some companies do an initial take-home "screen" just to filter to the candidates who are interested/desperate enough to invest in the "proof-of-work".
To reduce the jerky blow-off rate, are you getting at least a solid call with a manager or engineer (not just a recruiter phone screen) as a sign of their investment, before you invest in the take-home?
Balanced dynamics and mutual investment are good.
That’s why i really like my current company where we don’t set unrealistic expectations to candidates, we let them solve some basic coding task live and mostly talk to people. If the person can write code, a basic sql and clicks with everyone, we hire them. And our tech recruiter does wonders properly filtering cvs and recognizing fakers by just talking them. That’s probably the most important stage in the pipeline.
EDIT: Take-home-styled coding exercise in a pair programming setup is fine --- I've taken a few of those and they were actually quite pleasant. The task is not the issue --- respect is.
Adjacent to that, they probably don’t want to open themselves up to an argument about your work, while you try to convince them to give you another chance.
That doesn’t make it any less frustrating though.
I used up vacation time to meet with them, I would expect some type of acknowledgement regardless of the outcome. I didn’t even care if they gave me feedback. Just give me a “yes” or “no”.
The company was Splunk.
Check out the Glassdoor interview reviews to feel better about pretty much any other take-home haha.
Ive done the "2-4h take home test" once. It was blatantly clear they wanted a day or 2 of free labor, with no real feedback. So, no. I'm not doing that. You want me to do a day's labor? Pay my standard rates.
You want these jobs more than they care about you as a candidate. If one of the companies that sent you a rejection had instead hired you, would you be insulted about how they treated the other candidates? No, you’d probably take the job and be happy you got it.
It’s a game and you’re each playing by your own rules.
Yes, it was close to where I lived so I didn't lose so much time or money but I would at least expect some kind of response.
As for take home assignments, your mileage may vary but up until recently I was receiving feedback from most companies. As time goes by though, I see more and more providing just a generic template response, if any.
- You aren't familiar with the domain and its working as intended, you're just bad at time management.
- You're not even familiar enough with the domain to know you're doing a bad job.
- You learned the whole domain and it was a good use of time.
In any case, slogging through a take-home you're not qualified for should warrant an introspection on your part. It should still warrant a response from the company if you submit it, and it sounds like certain companies have a ghosting problem. But it's not surprising that people who spend 10 hours on something that should take 2 hours are perhaps not coming up with good solutions and not qualified for the job they're interviewing for.
You can apologise and say "sorry but other companies have ruined it for you".
Good luck with your search!
This will tell you a lot about the company and if you should be working there.
How much are these “senior” and “staff” jobs paying?
Link the assignment for the applicant to the dev team to anonymously help with.
also put a copyright/license on the code you submit. publish it if you don’t get the job.
The reason take-home assignments can work well in my opinion is that it allows the candidate to show off how they perform in _real world_ scenarios - as opposed to live coding computer science toy problems. Interviewers can assess what code looks like that resembles the code candidates would likely write on the job.
What companies like fly.io are doing wrong: if you ask candidates to put in multiple hours writing code for free, the _least_ you can do is give them an interview to give the candidate feedback on their assignment, even if you're going to reject them. In case of rejection, it doesn't have to take more than 15 minutes of time from the interviewers (which is far less than the hours the candidate put in) and the candidate walks away with (1) valuable feedback that can help them improve and (2) a positive view on the company and their hiring processes.
What many companies don't understand is that your goal should be for candidates you reject, to still become "promoters" of your company.
Here is how we do it at Source.ag:
1. If the resume is promising, we always do a first screening interview with the candidate _before_ we even consider them for a take-home assignment
2. The candidate gets the take-home assignment after passing the interview, works on it and sends us back their solution
3. We schedule a technical interview with the candidate. This interview can basically have 3 formats:
a. the assignment was bad: we tell the candidate at the beginning of the interview we don't intend to continue with them and give them some personal feedback on the assignment
b. the assignment was great: we take time to ask the candidate about their solution, to elaborate on their tech choices etc. and generally use the interview to understand if their "talk" is just as good as their "walk"
c. the assignment was ok-ish: we use the interview to figure out what the candidate's engineering knowledge is and to what extend it complements what we got from the assignment. It could be that a candidate didn't perform super well on the assignment, but when talking to them we discover there is enough skill, knowledge and team-fit to work with. This happens mostly for junior-medior candidates, not for seniors.
This is a time investment for us as a company, but has gotten us good hiring results and a positive reputation when it comes to hiring practices.
1) plenty of companies who do not require that 2) those who require do not respect those who spend time on these assessments 3) even if you post about this everywhere, there is a very small chance you can change anything
There are other job search hacks out there that work x100 better than assessments. Can't share them for obvious reasons