All the time. 300-400k SLOC in C++. Legacy in the sense that there were no tests of any kind. Little-to-no documentation. Solo developer at the tiny company. Fix bugs and add features while keeping the system available to the tens of thousands of users.
A more recent example: here’s a patch for a critical feature we need. It was written over a year ago. The original author isn’t available anymore. You can write the code from scratch or try to resurrect the patch against master.
Being able to jump into a project and lead people towards some goal is definitely a skill for senior developer positions. Yes, you generally have a team you can lean on and have the ability to do research and all that. But how do you show that you can do all that in an interview?
Agree with the conclusion that a good thing to test for is for problem-solving.
The tech side depends a lot on what you’re doing. Although it gets ridiculous and organizations get lazy with this part. You don’t need to be white boarding graph algorithms for a junior web developer role. If your application is a social networking role and you’re interviewing a senior developer or architect? Definitely. They’re going to be teaching this stuff and need to understand it at a deep level.
Last week I was debugging some similarly uncommented, undocumented, Go code from 2020 written by lunatics that is also a critical piece of the system.
It hurts.
Where are these dev jobs where _don't_ have to figure out some mysterious issue in a barely maintained GitHub repo semi-regularly?
Heck, I have to debug the stuff that some idiot (me) wrote six months ago and it might as well have been someone else who wrote it for all I remember about how to debug it
Edit: apparently this wasn't written clearly enough that I was the original dev that I'm having words with???
IMHO the willingness to "dig your way down" to solve a problem at the correct layer (rather than working around a bug or missing feature in a lower layer by adding post-hoc ameliorations in "your" code) is one of the major "soft skills" that distinguishes more senior engineers. This is one of those things that senior engineers see as "the natural thing to do", without thinking about it; and fixes that take this approach work subtly to ensure that the overall engineered system is robust to both changes up and down the stack, and to unanticipated future use-cases.
And contrariwise, fixes tending not to take this approach even when it'd be a very good idea, is one of the central ways that a codebase starts to "rot" when all the senior engineers quit or are let go/replaced with more-junior talent. When, for example, every input-parsing edge-case is "fixed" by adding one more naive validation regex in front of the "opaque, scary" parser — rather than updating the parser grammar itself to enforce those validation rules — your codebase is actively evolving into a Ball of Mud.
Of course, the tradeoff for solving problems at the "correct" layer, is that you/your company often ends up having to maintain long-lived, trivial, private hotfix-branch forks of various ecosystem software you use in your stack. More often than not, the upstreams of the software you use don't see your problem as a problem, and so don't want to take your fix. So you've just got to keep it to yourself.
(Funny enough, you could probably trivially measure an engineer's seniority through a tool that auths against their github profile and calculates the number of such long-lived trivial private hotfix branches they've ever created or maintained. Much simpler than a coding challenge!)
Jumping into an unknown codebase (which may be a library you depend on) and being able to quickly investigate, debug, and root cause an issue is an extremely invaluable skill in my experience
Acting as if this isn’t useful in the real world won’t help. The real world is messy, documentation is often missing or unreliable, and the person/team who wrote the original code might not be around anymore.
> Fix bugs and add features while keeping the system available to the tens of thousands of users.
Don't tens of thousands of users warrant more developers? Or having enough of a budget to work on tests, or other things to improve the developer experience and be able to work without lots of stress? That's unfortunate.
So not necessarily any budget for more than they did.
I thought the page was satire at first.
> All the time.
I guess the minor difference was that it was part of your paid job instead of a huge time sink to get hired.
This is the biggest issue I have with all these tasks is that they take precious time that could be utilized better. In my case: working on real open-source software instead of throwaway made-up problems. (Source: I had to spend 3 days to write a transaction engine toy once and then I didn't get the job for another reason.)
It wasn't that hard if you know the fundamentals (http, ssl). It was sorta fun and I got the job.
Imo it's more important that you can break down an existing best-in-class algorithm for x task and have your prospective dev answer questions about it/how they would use it, etc. Expecting every senior dev to be a best-in-class top mathematician as well as having a handle on the ever-changing language/library/ops/etc stacks is just crazy.
I'd never expect a senior dev on my teams to just know something like that, but I would expect that given a task they can do their research, find good candidates and understand the pros/cons of each approach. Because our power is in our ability to learn & use; the stuff that we already know is a bonus.
Not just all the time — right now in the other browser tabs I'm ignoring in favor of this one lol
So, definitely, I always need help debugging any kind of code (unless it's rather code that doesn't deal with product features).
You can, to a degree. If you can get the system under test you can make an assertion about how you think it works and see if it holds. If you know what the "law" is you can test whether the system calculates it according to that specification. You will learn something from making those kinds of assertions.
Working Effectively with Legacy Code by Michael Feathers goes into detail about exactly this process: getting untested, undocumented code people rely on into a state that it can be reliably and safely maintained and extended.
Depending on the situation I often recommend going further and use model checking. A language and toolbox like TLA+ or Alloy is really useful to get from a high-level, "what should the system do?" specification down to, "what does the system actually do?" The results are some times surprising.
You're right that at some level you do need to work with someone who does understand what the system should do.
But you can figure out what the system actually does. And that is de facto what the business actually does... as opposed to what they think it does, logic errors and all.
A good senior programmer, in my opinion, thinks above the code like this.
Update: interviewing for this kind of stuff is hard. Take-home problems can be useful for this kind of thing in the right context. What would you do differently?
agnostic programming flueny/software practices, including being comfortable debugging unfamiliar things or navigating uncharted waters without many domain experts to guide me, is the number one skill required in my role.
bonus points if i end up becoming an active maintainer to the project, or someone who can help other devs w it, the more times i dip into it.
Surprisingly, both jobs were fine too.
If you didn't solve your problem in 45 minutes with zero assistance from tools or Google or colleagues, were you immediately fired?
I mean, even if you have a team, you are hired to do specifically this kind of work. I'm confused what OP thinks a software job is if not EXACTLY this.
> All the time.
So you were forced to do it in a silo? No access to any type of documentation, whatsoever? Not able to refer to any C++ manuals or any other type of documentation? Not permitted to access the internet to look things up? Barred from using online forums to seek help?
The OP says: "A “4-hour” assignment". This is take-home, so candidates are free to use any C++ manuals, documentation, AI, online forums, whatever...
A partner of a friend quit their job earlier this year. They then took 4-6 weeks to prepare for each interview with Big Tech companies (4-6 weeks for Meta, 4-6 weeks for Stripe, etc.). Along the way, they also took random interviews just to practice and build muscle memory. They would grind leetcode several hours a day after researching which questions were likely to be encountered at each Big Tech.
This paid off and they accepted an offer for L6/staff at a MAANG.
Talked to them this week (haven't even started the new role) and they've already forgotten the details of most of what was practiced. They said that the hardest part was studying for the system design portion because they did not have experience with system design...but now made staff eng. at a MAANG. IRL, this individual is a good but not exceptional engineer having worked with them on a small project.
Wild; absolutely wild and I feel like explains a lot of the boom and bust hiring cycles. When I watch some of the system design interview prep videos, it's just a script. You'll go into the call and all you need to do is largely follow the script. It doesn't matter if you've actually designed similar or more complex systems; the point of the system design interview is apparently "do you know the script"?
Watch these two back to back at 2x speed and marvel at how much of this is executed like a script:
One of my hires was a physicist who didn't know any of the jargon and had a look of terror when she first started on our whiteboard problems. Once we led her to a comfortable spot and she got into it she started talking about tools she made to help with her physics work with zero dev knowledge (she was shocked when we called her tools software, she was like, but it wasn't REAL software, we were like hate to break it to you, but that was not only real software you wrote, but crazy impressive). The white board problems were a tool to highlight potential.
I get that these big companies just want drop in widgets not people and that is what they are searching for, I just think it's funny they are using something that was created for the exact opposite purpose.
Part of me is kind of glad I'm too old to be one of the cool kids and have no hope to land a job that does these sorts of code challenges.
Of course 95%+ of it will usually be useless in your real-life work, to solve most of those problems in the time given you need to know the problem and the optimal solution to it so basically memorization with little thinking.
In other words, anybody, regardless of what university they went to or what courses they took out what advantages or disadvantages they had, can learn this stuff in a couple of months if they have what it takes for the role. And because the skills are so standardized, the process is pretty differently objective.
It's not expected that they'll actually use the specific skills in the job. But it shows they can learn skills of that type and then perform them at a high level in a stressful interview situation. Which is a great signal for whether they can learn the skills needed for the specific project they wind up on and perform in a high stakes deadline scenario.
I'm not defending the system, but I am saying there's a clear logic behind it.
I know one manager who had issues with all three got hired at Google. So. Think of the poor HR person that will have to clean up that mess.
I've interviewed people with leetcode questions where my co-interviewers made the candidate's confidence in presenting the correct answer the tipping point for hiring (with women mostly being targeted in this category). Bias can happen in any process, and with "culture" frequently being a consideration it's hard to tell what should be justified. Meritocracy in the tech industry is mostly a joke outside the overachieving outliers.
Have you worked with them since they went through this regiment? Doing a DS&A coding problem regiment + system design will change you as an engineer. You might be surprised how good they've become.
Also they say they've forgotten. But if they were able to get that position, they probably could do medium level leetcode problems. So, I'd doubt they've forgotten all of it. I'm pretty sure they'd still be able to solve easy level problems, which most people can't solve off the cuff. They also probably still know the complexity of a bunch of essential backend operations (search, sort, array and hash lookups, tree & graph traversals, etc).
The point is not that the friend didn't pick up some implicit knowledge or become a sharper engineer than they were before grinding, it's that by exploiting the screening strategy, they got placed into a job they're not truly qualified for.
Are they bright enough to fake it until they make it? Maybe, but that's not going to be the case for many of the countless placements that were made like this, and hints at why both product and software quality is in bad shape these days.
Having done this prep, I can't tell if this comment is sarcasm. Building real systems makes you a good engineer. Maintaining systems over a long period of time makes you a good engineer. Working with other experienced engineers makes you a good engineer.
Doing this prep you do learn a few things along the way, but it's contrived. You already know what you need to learn, which is often the hard part on the job. It's works as a filter since the ability to learn concepts is important, but it's usefulness continues to trend downwards as it becomes more standardized.
You know you've designed more complex systems. The interview has no way of knowing if that's true, or you're an actor following the "I've designed complex systems" script.
And acting talent is arguably more common among the population than programming talent.
When I've hired or been a part of a hiring process, I always place emphasis on a candidate's past projects and ask deep technical questions around those. I also always review their GitHub repos if one is provided in the profile and I will ask them deep questions about their repos, why they chose this tech or that, interesting pieces of code, design tradeoffs they made, etc.
The bummer is that you're right, it actually is worth even this much investment, because these companies do pay extremely well. But it's still horrendously inefficient, because the companies are getting a very small improvement to their signal to noise ratio, at this great cost (which, notably, they don't bear).
The problem with this story is that people need to work with and rely on this person for responsibilities they're not yet qualified to meet. In some cases, a "smart person" will be able to grow into the role, but the road there is long and messy, which becomes a frustration for colleagues, supervisors, clients, users etc.
Because of the prolonged boom we just went through, the industry -- especially at FAANG's -- is now saturated with smart, naive people trying to fake it until they make it, leading to a gross decline in quality and consistency compared to where we have been and might otherwise be.
In my mind that's a rather nasty practice.
Not that I'm complaining. I'm happy to pick up people that are good at computers but wouldn't be able to pass that hurdle, and probably wouldn't hire anyone that has.
if you provide utility to the market, you shouldn’t need credentials or a corporate ladder to climb to prove it, it should be immediate compensation no matter how disparate
so despite how coveted tech compensation packages are at this tier of company, a seemingly smart person getting a good job after doing a contrived aptitude test does meet that criteria. other people outside the field (and within) have difficulty passing it and don't have the cognitive ability to study for it, or the financial stability to prioritize studying for it
I also agree that a less contrived aptitude test would be better
Your friend is not alone, and I don't understand what separates us. Even when I was early career, system design was my favorite.
My biggest issue is performance anxiety when there's a concrete problem to solve that I've never seen before, so coding sessions are absolute hell. Inability to focus, forgetting approaches I've known forever, etc.
Ask me to do system design though, and it's generally freeform, I can talk about different approaches, deep dive on the fly, and draw a nice diagram or many nice diagrams, and interviewers love me and think I'm smart. I've had interviewers who were suspicious after my bad coding performance try to nail me on esoteric implementation details but because I have actual, hard-earned experience I can pretty much explain any approach in my niche and then tell you all the tradeoffs. I can also quickly code up examples, etc. Never had a bad feedback on this portion.
Something about the complete detachment of the coding problem from real life is such a huge mental block and I am continuing to fight it 15 years into my career, after dozens of similar interviews.
People generally don't even want you to go deep on anything, but you're can't stay to shallow.
It's just bsing, yes your absolutely right, it feels like memorizing a script and saying the right words at the right time
The most surprising thing when asked about their experience was that most of the details have already been lost before even starting the role.
Demonstrating the ability to learn, even if it's just scripted, should be good enough. Granted, they might get passed over by someone with actual experience, but the lack of real experience shouldn't be a deal breaker. It's fine as long as the employer has realistic expectations of the employee.
Interviewers are just trying to find someone who would design a system the same way they did. They used a queue? They want you to add one. They used serverless? Better say the word "Lambda" somewhere. They hate serverless? Ooops better stick to regular servers.
In the end yeah, it's about following a script, but also reading the room.
If he was even considered for L6 staff, then his on resume credentials already justified him at L5 Senior or L6 Staff.
The interview performance is what pushed him over the edge into L6.
It seems like you think they don't deserve a staff level position?
> If he was even considered for L6 staff, then his on resume credentials already justified him at L5 Senior or L6 Staff.
1. Do a takehome test, targeted to take about 4 hours but with no actual time limit. This was a non-algorithmic project that was just a stripped-down version of what I'd spent the last month on in actual work.
2. Do an onsite pairing exercise in 2 hours. This would be a version of #1, but more of "see how far we get in 2 hours."
3. Submit a code sample of pre-existing work.
Based on the ire I've seen takehome tests get, I figured we'd get a good spread between all three, but amazingly, ~90-95% of candidates chose the takehome test. That matches my preference as a candidate as well.
I don't know if this generalizes beyond this company/role, but it was an interesting datapoint - I was very surprised to find that most people preferred it!
This is a key thing for me. I consider it a moral obligation to give material feedback to anyone who does a takehome test for me - to invest at least some serious time evaluating it. Likewise, I would never give a takehome test until the candidate has at least had a phone screen with someone at the company and there's some level of investment on both sides.
On the other hand, I know a few junior devs right now who are submitting resumes and getting sent takehome tests off-the-bat. And, of course, after spending hours on those challenges, they get only form-letter rejections. I understand why companies do that - there's a glut of junior devs right now and any early-career role gets flooded with resumes that you need to somehow pare down - but I still consider it unconscionable.
I understand why you might not want to risk dealing with that.
I'd favor 1 and 2 over 3, because I feel that you understanding the problem I'm solving gives a better context to appreciate my approach.
If I take it home and hand it later, you can compare my solution to yours (or others) and better grasp my coding style, even if it's different to yours, since we worked on essentially the same task.
If we're working as a pair, you can observe my thought process. I can better express my reasoning for picking certain idioms.
With past code, there's almost no context. It's just code that was written for some other project. In my opinion, this can be very useful in preliminary steps, when I'm sending out my resume and you need to validate that I can indeed write code. In later stages, I think it can be impressive if I have a working app and can walk you down the code for a particular feature. But who has working software laying around? We all have code that works with a few hacks, sorta...
I'd favor 1 over 2, because it reduces the probability to blunder. Between spending 2 hours on-site with a higher risk to mess up (due to pressure, Murphy's law, etc) and take the assignment home to work on it for an extra 2 hours, with ample time to freely research whatever I don't understand. It's a no-brainer for me.
When I hire I don’t need someone to impress me, he has to present himself as able to do the job.
I think a lot of devs think they have to hire someone that impresses them and we end up with insane tests and adversarial interviews.
Yeah of course there will be people who will blow through the task much faster than others with better quality than average and they will impress me and they will get the offer in first place - but often times they already get multiple offers and after negotiation they will pick some higher offer and we have to get someone we can afford to do the job.
I was expecting "3. submit a code sample" to be the overwhelming winner here - did anybody choose that? Seems like a no-brainer, since it's already done...
On the employer side, I would prefer the take home assignment over existing code, unless the existing code was super high quality or highly relevant to the company.
As others have pointed out, there are a lot of problems:
- Many engineers don't write code outside of work and so don't have much code they can show off without breaching their employer's trust
- Those that do often don't have recent code.
- Those that have recent, personal code to show off have often written it to accomplish some goal and the code isn't necessarily a great example they feel like showing off
Really, the only people I've seen have good code samples are those who do extensive open-source work. And for those people, I'm probably already aware of that work when I checked their resume + opened their github.
Last time I did a coding interview for real, I had the choice of any programming language, and could choose between 3 different problems to solve. I liked that quite a bit, and was offered the job. Being able to choose Python, instead of, say, C++ in a time-bound interview almost feels like cheating.
That makes sense, and it's the perspective that's being drilled into a lot of us. Implicit bias and all that. But in my experience, the comparison problem has never been that big of an issue in practice. I guess it depends on the hiring climate, but I'm much more familiar with spending a lot of time going through mediocre candidates and looking for someone who's "good enough". And this is when the recruiter is doing their job, and I understand why each candidate made it to the interview stage. Sure, sometimes it's a hot position (or rather, hot group to work for) and we get multiple good candidates, but then the decisionmaking process is more about "do we want A, who has deep and solid experience in this exact area; or B, who blew us away with their breadth of knowledge, flexibility, and creativity?" than something like "do we want A who did great on the take-home test but was unimpressive in person, or B whose solution to the take-home test was so-so but was clearly very knowledgeable and experienced in person?" The latter is in a hypothetical case where everyone did the same stuff, just to make it easier to compare, but even in that setup that's an uncommon and uninteresting choice. You're comparing test results and trying to infer Truth from it. Test results don't give a lot of signal or predictive power in the first place, so if you're trying to be objective by relying only on normalized scores, then you're not working with much of value.
Take home tests or whiteboard tests or whatever are ok to use as a filter, but people aren't points along a one-dimensional "quality" line. Use whatever you have to in order to find people that have a decent probability of being good enough, then stop thinking about how they might fail and start thinking about what it would look like if they succeed. They'll have different strengths and advantages. Standardizing your tests isn't going to help you explore those.
Now add this to the fact that a vast majority of your applicants are going to be feeding your assignment directly into an assistive LLM.
As an interviewer, you really can't win.
- If you have a take-home test, people will either complain that it's too involved, and time consuming.
- If you do whiteboarding, people will claim that it's not representative of the actual job.
- If you do paired programming, people will claim it's unfair to those who don't test well under pressure.
The fact is, it all depends on the person you need and the team they will be slotted in. But it seems that the ones who know the criteria is rarely involved in these matters.
As for feeding it into an LLM, I go back and forth on that. With their current capabilities, the project is slightly too complex for that to work. You could get a lot of help from an AI, but you still have to put the pieces together yourself. And I'm fine with that! If AI tooling makes you a faster/better dev on a test, I'd expect it to do the same in real work.
Longer-term, though, I worry that LLMs still won't be able to "write the whole thing" for real work, but will be able to "write the whole thing" for takehome tests. And as such, I'll have to figure out how to handle that.
I used to think like that ... then we hired a batch of 3 "senior" devs who could not do simple, everyday coding tasks, even in their ostensible languages of preference. All had come with exceptional resumés and personal recommendations.
So now, no one at any level one gets hired without demonstrating that they can at least write a fizzbuzz and commit it to a repository.
Now, I doubt that senior engineers in other fields have to do this sort of thing. But, most other engineering fields have licensing/accreditation with accompanying post-graduate academic curricula. We don't (but probably should).
I exclusively give pairing interviews, usually just 1 hr. The variance between good and bad candidates is amazing, and you wouldn't guess at all from resumes or casual conversations. Most candidates have given me very positive feedback. Why wouldn't you want to be interviewed like this?
https://www.researchgate.net/publication/283803351_The_Valid...
If you've actually worked at a large company, you'd know that 90% of the real work is done by like 5% of people (maybe even less). If the interviews worked this ratio would be so much better.
"I want to work for your company"
"Ok, prove yourself"
"Sorry, I don't have time"
How do you expect that conversation to go from there? If you don't have time then make time. It isn't anyone's problem but your own.
- Cracking the coding interview.
- Elements of programming interviews in Java|Python|whatever...
- leetcode & other sides with paid premium subscriptions...
- mock interview bootcamps...
It's no longer about skill, it's only about gaming the system.
LLMs are another great example
I’m serious. It’s not even guaranteed you’ll get the job if you do the above but everyone who passed the big tech interviews will acknowledge they fucking studied for it. What do expect here?
The companies ask that candidates learn how to solve algorithm problems and the candidates do it.
I would call it “everyone playing a game they agreed to play by the rules they agreed to play with.”
And I don’t know what you mean by “it’s no longer about skill.” It still takes a lot of skill to be able to solve hard algorithm problems, even if you took a course on how to solve them and practiced solving them for 6 months.
When you audition for an orchestra they give you the sheet music ahead of time. It doesn’t mean you have no skill if you practice first instead of just showing up to sight read the music.
TFA: Coding interviews are a standard way of hiring.
Also you: “Too many people in the software development industry don’t know what they’re doing.”
The lack of cognitive dissonance is remarkable.
GP is of the opinion that coding interviews filter out people who can't code.
How can we even ban them using LLMs in their "ML-leetcode" problem when the problem is on LLM tokenization or something?
Like, if I lose a candidate because they're experts at using cursor + claude to do their coding (i.e. they solve it by writing 3 prompts and filling in 2 errors which are easily spotted in a total of 10 minutes), despite them having NeurIPS caliber publications and open source code, did I filter out a fake grifter, or did I filter a top tier candidate?
I just don't buy that leetcode style problems make any sense for anyone whose doing anything involving LLMs, and like it or not, increasingly large amounts of the cushy 500K/yr FAANG+ jobs will involve LLMs going forward.
outside the geography where the salaries are an order of magnitude less, we yet go through the pain because of companies copying each other's approach.
And frankly, if they're able to adapt a random leetcode problem to be able to solve the coding test that I give them, then that's exactly the kind of adaptability that I'm looking for in a prospective software engineer.
Sounds like an OK test to me. Great (senior) developers should be able to do that kind of thing. Categorizing yourself exclusively as "a Ruby developer" is a career trap.
And a lucrative one at that.
It's ridiculous how developers mindlessly accept that you should constantly be learning to keep yourself relevant, but keep it shallow by just jumping from one tool to another, instead of encouraging deeper knowledge of generalizable patterns that stay relevant across waves of technological disruption.
No engineer that makes 7 figures calls themselves a ruby developer with the exception of DHH.
Debugging old DSL vendor specific languages or code so old using, frameworks and standards long out of fashion and support, that they are half way to being a different language.
Adding support for some back ported features or patching security holes in an old client or legacy stacks.
Or at a big company we had some escrow code from a much smaller partner that we ended up becoming responsible for.
Often getting the environment setup for proper debugging is more work than anything.
But yes, it's a good test for a senior+.
Interviewer: Now reverse this array.
Me: OK, in Python that would be array.reverse(), or reversed(array). I bet JS has one of those, probably the .reverse method.
Interviewer: Great guess!
That was genuinely fun. I came out of it feeling like I'd learned a few things, and the other person got to see how I'd reason about a new problem.
It was a pair programming exercise and so with some help from the interviewer and the IDE I was able to fumble through to a working result. I agree it was fun and educational.
Know what I'd do if the interviewer asked me to debug PHP? Pretty much return the question:
"I've never used PHP. Are there logging macros/functions defined somewhere? Where do I see the output? syslog maybe? Is there a debugger of some sort I can use? How do I run each `piece` of code in isolation?"
(I am assuming the job listing did not explicitly mention PHP experience. If it did, both myself and the recruiter would absolutely deserve to fail me for this interview).
I have had to do this, on numerous occasions. Sometimes there is a system that no one who built it is left, the documentation doesn't exist or can't be found, and that's it. This is not a great example. Being able to debug a program, follow requests around and see what's happening is a good skill.
What this example is really bad at though is that you pigeonhole yourself into developers. If you have a legacy php application, you are weeding out the java, python, ruby, go etc developers that do have those skills but don't know php and will be slower at it. Despite them possibly being stronger developers given a couple months of hands on php.
> A “4-hour” assignment can quickly turn into 8, 10, or even more, just to ensure it’s in great shape.
This I agree with. Suggested time can easily be gamed, so someone who has more time can look like a better candidate.
Literally every day this week.
Documentation? The code was last updated in June, the documentation was last updated in January - of 2022.
The team? The half that didn't get laid off two years ago all quit for greener pastures. Sure, there's a team that owns this project - along with three others, all of which they've contributed maybe a dozen lines to over the past year, and 11 of those were dependency updates.
> This is like asking a Ruby developer to debug PHP as a test of flexibility
> If the job requires specific tech skills, test those skills
Sounds like someone failed a coding challenge.
In all seriousness, you need to understand that companies rarely implement their hiring practices for the sake of it. It is usually the best their collective minds could come up with. Telling them to get rid of it without attempting to understand what problem they're trying to solve and without offering alternatives, is, to say the least, not a productive use of anyone's time.
I find in software development we try to reinvent the wheel all too often instead of borrowing practices from other professions. Let's have a look at what civil engineers have to go through to get hired at half the salary of a software dev, and let's incorporate that into our practice. That will make everyone a happy trooper !
Unlikely. It's usually what the most senior person either read about or experienced in the past. In fact I'd say that it is highly unlikely that they did any introspection at all as to what they are actually trying to accomplish.
They just did it like everyone else because they believe exactly what you do -- that someone somewhere created the process with intention.
I feel like we've so lost the plot with tech hiring that we settled on what appears at best to be a local maxima.
I'm told this is still the common style of interview for mechanical engineers, which says something about what it's like to work in that industry too.
Based on the observations I've made of hiring managers I've worked with, what they're trying to accomplish is to provide the appearance to HR that they at least attempted an unbiased hiring process. The result often resembles a "literacy test" and I suspect one has to be very intentional about avoiding that to practically avoid it.
The issues began when we started working together in a Data Engineering team. Most of our stack was based on Hadoop, Kubernetes, Ruby, Python, and several other technologies that required a basic understanding of cloud computing.
However, since we had such a wide range of work experiences and backgrounds, I often found myself not doing the work I was hired for but instead covering for others. I ended up doing a lot of "glue work" to compensate for the fact that some colleagues couldn’t even handle basic tasks, such as Python CLI packaging using Docker or inspecting jobs in Hadoop. After some time, I decided to leave—not because I thought I was special, but because I was fed up with spending 80% of my time providing support for people who were hired at the same level or even higher than me.
I recognize that the company had very poor hiring practices, but some form of basic technical testing is necessary.
They were unfit to complete the tasks. But again was not their fault, I think was the hiring that failed.
One concrete example was the fact that our pipelines was quite straightforward for data engineers: packaged Ruby and Python CLIs that runs commands. The runtime was k8s. One of the biggest issues were when something broke in production, none of the people couldn’t go to the container, check the logs and understand the failure.
There’s one situation where I think makes sense to hiring without a test: if the company has the resources and money to provide levelling training for all new hires with no exceptions. I do not know how practical it is.
The fun part of take-home assignments is that there's a moral hazard incentive to spend extra amounts of time on an assignment even if they say "only spend X amount of hours on it," even lying amount the amount of time spent if the interviewing company asks.
More than once I have completed a take-home assignment with emphasis on sticking in the timeframe they recommend but still solving the problem adequately, and then I get chewed out in the followup interview "why didn't you implement X?"
Relatedly, a massively open-ended take-home assignment is a bad take-home assignment since it favors those who have more free time to be thorough. (shoutout to the take-home data science assignments which ask for the final deliverable to be a PowerPoint presentation)
We are blessed.
It's not a perfect system, but it's a lot harder to fake being an experienced doctor than it is an experienced developer.
With software you never have to stop proving yourself, and your skillset is always a few years away from being outdated. A doctor with 20 years of experience would be welcomed anywhere, but an engineer with 20 years experience is viewed with trepidation. The next "big thing" could roll out at anytime and suddenly crypto engineers are getting 800k job offers so everyone furiously tries to learn crypto stuff. A few years later that all dries up and now you have 100k engineers who are out of work and learned tech that no one cares about anymore. All the LLM engineers might be in the same position next year.
Of course. They paid an enormous cost and beat other competitors for that privilege which is carefully guarded by regulation and certification boards.
No doubt software has more instability, but the trade off is that you don’t have to do that multi year grind. Easier fire, easier hire. Similar pay.
> skillset is always a few years away from being outdated.
I strongly disagree with this. In your example you use “crypto” and “llm”. If this is your skill set you are fad chasing and needlessly increasing your exposure to changing markets.
Engineers solve problems by applying math and science expertise. This is a timeless skill.
Didn't have to solve PDE's on the whiteboard, or regurgitate integral transformations.
The insane thing would be to expect someone to take you at your word for 150k*, when hiring managers know that 50-70% of people with your resume fail to write a nested for loop.
I’d say yes and no.
Yes, these are the problems that cannot be solved perfectly.
No, because in such areas any ‘reasonable’ filter is better than nothing. People say that these assignments don’t have anything with reality, but, well, we don’t have months to try to work with each other, we only have 3x1h.
I worked as individual contributor for years, but also had a chance to try a hiring manager role for the past 3 years a lot. We do standard leetcode-style interview (without hardcore) + system design. And I always consider both as a starter and bite to see how the candidate behaves; talks; do they ask questions to clarify something and how. And I always try to help if I see that candidate is stuck. By the end of all interviews you will have some signal, not a comprehensive personality profile. Do we do mistakes? I’m pretty sure, yes. But I think it just works statistically.
1. Applicants getting hired because the manager and the candidate is from the same institute or they like the same team or band.
2. Bias based on age or race. I try to remove my own sub conscious bias as best as I can but most don't.
3. HR thinking you are not good enough and rejecting because you did java 18 and not java 19. Or you can write impressive React but haven't done Vue.
It takes the managers and HR a little out of the equation. They just rubber stamp your decision.
Take home exercises are something we tried but the incidents of cheating were so high and good code is so subjective that we gave up.
If I were interviewer, I'd probably do a coding assignment that builds on an already existing mini codebase, quarter or half day assignment (2-4 hours), with a strict time limit to turn it in by the end of the allotted time. It would test for someone's ability to read unfamiliar codebases, with the codebase being reasonably small for the little time that's available, and the ability to build features on top of it, to write clean readable code compatible with the codebase, etc. And an a 30-60 minute more open-ended informal knowledge interview to gauge how knowledgeable the candidate is and where their strengths and weaknesses are. And previous work would probably be a good indicator, like open source work and/or previous job experience.
The problem is not coding challenge per se, but that people can now cram it on sites like leetcode. See, before we had leetcode, only two types of people could solve a large number of algorithmic problems organically: those who were naturally talented, and those who were so geeky that they devoured the works of Martin Gardner, Knuth, and the like. Excelling those challenges showed raw talent in old days.
And companies like the young Microsoft and young Google absolutely loved such talent, and such talent did shine in those young companies.
Not that I can pass the interviews, by the way, and I definitely don't want to prepare for such coding challenges. I'm just trying to explain the phenomena as an observer.
It depends on what kind of functionality we're talking about, but this kind of task is exactly what people at my current startup have been assigned at times. It is absolutely possible to build a CRUD web app with reactive UI using modern tools in a few hours.
> This is like asking a Ruby developer to debug PHP as a test of flexibility
Again it depends on what the debugging task is. At every startup I've worked at, it's expected that an engineer is able to jump into a task that they know very little about. Granted it becomes less reasonable the more niche the task, but PHP and Ruby are not particularly far apart in skillsets in the grand scheme of things. I would expect any web engineer to be able to do this.
> Hiring processes should focus on problem-solving, collaboration, and growth in relevant areas
I agree with this. And, hiring should also focus on technical ability which does include working through difficult and unknown problems by oneself.
Yes, but it also leads to tons of bikeshedding. "Should I implement a linter? Should I write unit tests? Integration tests? Should I mock HTTP calls or use a mock server? How should I deploy things? How do I structure my CI?" All these choices are extremely subjective and context dependent and then somebody's gonna pass on you because they don't like that you used Cucumber.
Setting up a project is not something you do often, and if you do, you probably have templates and some team standards.
It's much better to give somebody a project structure and ask them to implement some basic task in it.
But how often do devs normally set up a project from scratch? We have like 3 new apps in a few years where I work now in addition to the long running ones. So on average one in like hundred devs here have been part of creating something from scratch.
Sure, when you know what you're doing it's quick. But the first time can be slow, no matter how good you are. So for a coding task that should take a few hours, you might have spent all the time just getting up and running, and then you actually start the task as over time.
I've had to do this at some level in every job.
Somebody left. The code was left to rot, whether it was a separate codebase or it was an entirely separate project. I got assigned to fix it. Sure, I had a team, but nobody knew more than I did, so it was just other devs who might have another perspective.
Collaboration and support might be standard, but mostly in the form of other smart people, not always documentation, up to date software, or people who know anything about the code.
I personally prefer these kinds of challenges as they mean I don't need to maintain two skillsets.
Yesterday! Working on a large project means there’s always some issue with a dark corner no one has looked at recently and because there’s no team to support, I get to go and bug hunt.
Today, privilege means starting from no real job experience to writing enshitified_prod_tokenizer() and laughing at the poor sap having to fix it in a few years.
But having a simple coding task does some great things in that it will make the interviewee have to fix problems. They need to ask when unsure. You'll know who stops and asks and who plows through and solves the wrong problem. You'll know who throws their hands up when instructions are unclear and just says "I can't do this" and gives up. It's one of those extremely valuable and very effective filters that you just shouldn't be without.
The actual assignment can be however simple, but if they manage to install the tools, clone your repo, read the instructions, build the thing etc. then you already filtered out half the applicants or more. And that's without even starting to code.
Sure, how? There are no perfect processes, only drawbacks that you can live with.
I also find the idea that interviewing is somehow exploitative of the interviewee ("feel like working unpaid shifts for a job they don’t even have yet") to be somewhat bewildering. It's not like your coding challenge is useful work for company that you are doing for free, in most cases interviewing is also a huge time sink for the company as well. Not every interaction should be quid pro quo...
The hiring manager might hd really liked a candidate and conclude that a coding challenge just shows that the new hire would need extra time to ramp up.
Yet, ignoring money, you could excel in the coding part and the hiring manager might not like you because you lack "X" (e.g. "communication skills", "team culture", etc), which in reality is a way to put they just didn't like you or there were not really looking for someone to deliver, example, they were looking for someone to play politics or that might benefit the hiring manager more in the long term.
Not always the more technically capable candidate gets the job.
This is pretty much my day job. OP is spoiled if they think this is an uncommon situation.
Can people game them? Are they flawed? Absolutely. But that's true of literally any interview practice.
The idea behind whiteboard interviews is if they're challenging enough, the only people who can fully exploit them are either very bright, very motivated, or some combination of both. That's a better signal than asking someone to lie about their "passion," which any slob can do.
Pretty much all the time. I also found that debugging things without asking for too much help was a great way to learn the systems involved, which eventually made me a better and more useful programmer, with more domain knowledge.
Granted not everyone has the luxury to spend time doing this, but I don't think it's that rare. In many cases it's even what companies want you to do, when that documentation or help is simply unavailable.
Here the article suggests "Hiring processes should focus on problem-solving, collaboration, and growth in relevant areas". Ok, but how? Problem solving is typically tested using puzzles of some kind, like... coding challenges, but apparently, it isn't the right way, so what is the right way? And how do you test collaboration before hiring? The hiring process is competitive by nature. As for growth, again, hard to test in advance.
Suggestions I have seen are:
- Interviews: great in theory, but some people are great at bullshitting, others are competent but just don't do well in interviews
- Open source contributions and past projects: great if you worked for companies doing open source before, not great if your past work is confidential
- Assignments: Hopefully paid, but in any case, very time consuming
- Puzzles, IQ tests, school-like knowledge tests, ...: like coding challenges, but worse
- Probation period: hiring someone just to fire him a week later is not great for morale
- Dice rolls: maybe worth considering
Coding challenges have a number of advantages. They are unbiased with regard to age, race, gender, religion, etc... They test a skill that is at least related to the job. They can be done remotely and don't have to take that long. Whatever replace coding challenges should meet these criteria too.
"Are you testing the candidate on what the job will actually require, and ONLY that?"
With an onsite whiteboarding challenge, you're most likely testing if they're really good at thinking and talking in front of a crowd when extremely nervous, not writing code.
Same with a live coding challenge over zoom (CoderPad style) - this tends to be a test of nerves more than coding ability. But, if you're planning on a lot of pair programming, and want them to be able to dive into that on day one, maybe it's the right way to interview.
I find take-homes often the most accurate way to interview, as it most closely reflects the reality of most coding jobs. You're on your own, you don't have to deal with the extremely odd (and for some of us terrifying) feeling of someone watching you type. Hopefully, the take home questions are carefully designed to reflect what the job requirements will be. As almost every other comment has pointed out, debugging an ancient code base without documentation may be a perfect question, or it may be completely irrelevant and unnecessarily eliminate candidates.
But please, make the interview match the requirements you're actually hiring for, otherwise, everyone's time is being wasted.
As a hiring manager, I strive to evaluate for potential and adaptability, not a highly specific list of skills. I think many candidates prefer that, too, that they will be allowed and encouraged to learn how to do this job. I do not want to hire people that don’t want to learn or can’t do things they haven’t done before. And I may or may not even know exactly what the job will require. Do not assume your interviewer can tell you exactly what will happen once hired — they can’t! One thing I do know, for sure, is that the job will require communication with others, adaptability to changing requirements, and a willingness to learn the company’s workflow. I need to make sure those boxes are checked, sometimes more than I need to see how well someone knows jQuery or CUDA or sorting algorithms.
* Some companies design take-home assignments that mirror challenges their team has spent years refining, setting unrealistic expectations for candidates to match or exceed that work. Talent is sometimes dismissed over minor differences, like testing styles or even coverage percent.
* Working with people is about compromise. When passionate people collaborate on a creative endeavor, there's always some back-and-forth of ideas. But in these assignments, often only the unilateral opinion of the evaluator matters. I guess being rejected from a monoculture early in the process may actually be a blessing in disguise.
* Many companies fail to provide specific, actionable feedback after candidates spend hours on assignments.
I don't mind take-home assignments, but I avoid investing excessive time in them, as it’s not sustainable and often has low ROI.
AthenaHealth in Watertown MA, this is you, you miserable pricks. Didn't even send me a "No thanks" email.
I’m speaking here as a developer who has done more than just bug fixes or UI tweaks. I’ve built apps that have served millions of users. I even have personal projects that currently serve 20–30k users daily, with over 1 million downloads and a 4.9-star rating from 41k reviews. Beyond Android, I have experience managing servers with Nginx, setting up SSL certificates, using Cloudflare, AWS, and Hetzner.
I’ve designed an app with users, posts, and likes, handling both the front-end and back-end using cloud functions and Firebase. From writing Firebase rules to encrypting everything, I understood and implemented it all within a few months. I’ve also written numerous web scrapers in Python for personal projects, which are still in use. While some backend concepts might seem basic, I believe knowing how the backend works is a huge advantage for an Android developer.
Despite all this experience, I recently failed to solve a "number of islands" problem in an interview and got rejected:
Given an m x n 2D binary grid representing a map of '1's (land) and '0's (water), return the number of islands. This left me feeling useless. It made me question whether I’m fit to be a software developer. I studied DSA in college, but now I struggle to recall or prepare for it. Does this mean I’m a poor developer? Does someone who grinds LeetCode just necessarily have better skills in general? Is this what companies want? Are they okay to spend money and time in training someone just grinds LeetCode for the purpose interviews. Does someone who does this just for interviews is likely to stay in same company?
I believe there must be a better way to evaluate candidates. It makes sense to test DSA in campus interviews, where students are fresh and studying it. But for experienced developers, especially for roles like Android development, DSA is rarely relevant to the job requirements.
I’m not saying I can’t practice and do DSA—I can. But it has never been a job requirement for me, and I doubt it ever will be for an Android developer or a similar role. If it’s ever truly needed, I’m confident that most experienced developers, including myself, can excel through research and applied knowledge.
#Kill-DSA
All I can say is that perhaps all those places that reject you are not actually the places where you really would like to work.
At one point I managed to get a job at a company which from the outside looked like a really cool place to work in. However all these interviews painted a very different picture of the work that I was actually assigned to do. I felt like I was being paid a senior engineer salary to do the job of a junior engineer. I was utterly bored and I complained that it's too boring and way below my skill level. This ended up getting me fired. Strangely, I've never had so many work-mates expressing to me how sad they are that I'm leaving. Several of them left the company not long after me.
Currently I'm in a happier place. Not getting as big of a salary but having an opportunity to work on a way more interesting software. And how did I land this job... after a single short interview they told me: would you like to start tomorrow?
That said, you are making some assumptions. They might be right, but don’t forget to consider the possibility that they’re not, and then think about how to validate your assumptions, and about what it means and how to respond or adjust if your assumptions aren’t right. Also don’t forget to think impersonally about whether the job is the right fit for you. The job with the islands question might be a job for which it will be important to know and write similar algorithms, and based on the experience you describe, it might not be the right job for you. You sound like a full-stack dev, or even maybe a devops or generalist kind of person, and any given job might be looking for a more specific algorithm developer. When the job isn’t right for you, it is not a reflection of your existing talents or abilities, it could mean nothing more than they’re looking for someone with different experience or interest than you have. Being rejected might be best for you, before you know it.
> DSA is rarely relevant to the job requirements
I don’t know what this means. Generally speaking, a basic working knowledge of data structures and algorithms apples to damn near everything in computer programming jobs. I’d agree that most jobs don’t require knowing the subtle differences between parallel mergesort and parallel shell sort, for example. But most jobs that involve programming do require knowing some data structures and some algorithms.
Furthermore, companies are usually looking for well-rounded candidates. If I can hire someone who does full-stack Android and knows a lot of data structures and algorithms, that is better for me than hiring someone who’s great a full-stack stuff but gets stumped if they have to fix a connected components algorithm. Companies want devs who are good at as many things as possible, even when they say they’re looking for something specific.
Keep in mind that you’re competing with others, not being evaluated by yourself on your own merits. Also keep in mind that a single job rejection is nothing in the big picture, many things can go wrong in an interview, so look at the big picture and not individual interview questions.
The weird thing is that while people keep complaining about the focus on algorithms in the interview context, it seems like nobody objects when it comes to the recommended CS education.
I think there is a deep dissonance when people strongly recommend learning data structures and algorithms as part of CS and software engineering curriculum, but then when employers actually ask about these in interviews there is strong pushback.
I personally tend to think the right balance is in the middle. At least, whatever is taught in a typical software engineering curriculum should be fair game for interview questions.
The “number of islands” problem could be something that comes up in a DSA assignment in college.
I don’t know whether not knowing how to code it up in an interview setting makes you a bad engineer, but those who still remember their college classes got ahead of you apparently.
I often argue and get downvoted in the discussions when I claim algorithms and advanced math isn’t really needed as a prerequisite in the software industry. It’s only when people get gatekept for forgetting things taught in their college classes that they complain. Imagine that.
Relevant part: the interviewer/CEO's philosophy is not to code challenge because no one properly vetted for "problem-solving, collaboration, and growth in relevant areas" (tfa) would ever accept a job they were going to fail at. Will always stick with me.
Now, here's the weirdest interview of my life: An acquaintance who runs a software business interviewed me by bringing me along to an NBA game he has season tickets for. The interview in the dining area before the game was normal, the game included some breaks for more interview discussion, but the home team had the best shooting night of their entire season so the crowd was very rowdy and we each had 2 beers.
If you are a hiring manager and would like to experiment with something like this, please get in touch!
$100 is insultingly low for a 4 hour assignment.
If we assume 15 days off that leaves roughly 245 workdays per year and with a salary of $200k that would be close to $800.
A fair amount of compensation to spend half a weekend on a project would be $400.
The question made sense: how do you add to very large numbers together? I assumed the key knowledge they wanted was that the numbers had to be represented as strings and you should use bc_math to compute them. So I answered that, and they said, "Yeah, great. Now write a function to do that." I sat there for a few moments thinking about it and realized it was just nuts. So I asked if I could just end the interview process. They were clearly shocked, I doubt many people end an interview process on the last question. I said that this wasn't realistic and that if I did that for real it would get code-reviewed to hell. One of them kept saying but it's to see how you would do it if bc_math wasn't installed. I simply stated "I would have them install it" to which one of the interviewers instinctively nodded since that's the real-world answer.
It seemed like it would have been a good place to work up to that point. I didn't like my current job at the time, it paid considerably more, had more public holidays due to weird German reasons, etc.
I thought it was a hard but fair question. I still feel bad about not doing well. It wasn't all that close to what I'd be doing in the position, but it wasn't unrelated. Perhaps if it were a PHP role I'd be more bothered? But even there, it seems like one way to check if someone understands what's going on under the hood, so as to not naively do expensive stuff without realizing it. The fact that one should not write their own library, and that if you tried you should get code reviewed to hell, isn't relevant if that's the purpose.
But really, the fact you should never write one should be very basic and fundamental knowledge. Quite simply it should be an automatic no if someone didn't know they shouldn't write that themselves. They were literally testing people on things they don't do. Which means everyone I would be working with wasn't tested on their ability to do their job.
If you're not willing to do coding challenges, your ability to write code and reason about your decisions probably suffers as well. It's self-selection.
We told candidates to not take more than 2 hours on our challenge and we meant it. I tested the challenges myself from scratch to verify that it was possible in 2 hours. I still only consider myself a quasi-engineer, so if I can get it done in that timeframe, a full-time engineer using AI should surely be able to.
There's a few considerations that I believe make coding challenges as fair as possible:
- Make sure the challenge 100% relates to a real task they would do on the job. As an example, for our engineers who would be writing open-source Python ETL integrations, our test was to build a Python ETL integration to pull episode data from a IMDB api.
- Tell the candidate that it's ok to not clean or refactor the code. Meet the requirements of the task first. Your job in the interview process is to ask them what could be improved or why specific decisions were made.
- Don't have anyone perform a technical test until you've first invested a decent amount of time into them as the employer.
- Ask the candidate how long it took them. If a candidate takes longer than the allotted time, they're showing you that they'll overwork themselves and not follow instructions.
Agreed. When I think about the best engineers I've worked with-- and the worst-- I have identified certain characteristics that I want to look for in a candidate, and certain characteristics that I must avoid.
In particular, there are certain people who just can't entertain any thought or design idea that they themselves did not have. This makes them very hard to collaborate with, except in the cases where what we need to do just happens to coincide with what they are thinking. Related, there are certain people who can only think about exactly one way to solve a problem, and there is not room in their brain to honestly consider any alternative approaches. These people tend to be really bad team players. A good engineer will understand that there's no one right way to do certain kinds of things, that everything comes with tradeoffs.
During interviews I present coding problems where there is more than one way to attack the problem. Whatever approach the candidate does, I propose an alternative approach and ask them to talk with me about the pros and cons of the alternative approach. I also ask for them to extend upon the alternative approach and make it better. For a significant fraction of candidates, they just can't entertain any other way of doing the thing, there just isn't room in their brain to consider alternative approaches.
Ability to really listen and give an honest consideration for other people's ideas is a vital part of the kind of team culture we need to succeed, so I look for that in interviews. Long ago I abandoned worrying about leetcode in interviews, and nobody in more senior management has made me stop yet, and this is at a big tech company.
For example, it's not uncommon for a challenge in finance to be "build an exchange matching engine that takes orders, modifications and cancels from different threads in two hours." That's totally doable if you just fly through whatever comes to the top of your head, and maybe have an hour left over to test, debug and modify it. But that project (an exchange matching engine) is something that the CME has 10 full time programmers working all the time on. A more realistic 2 hour task is something like "look at edge cases for the order cancel type, and figure out a way to handle malformed inputs." Then you can actually write production quality code for them to judge.
This is a feature not a bug. Companies are testing if you can focus and complete a hard uncomfortable challenging task, because at your job you’re expected to do things you don’t want to do, but will be rewarded for doing.
That is an option if you have higher personal obligations and are struggling mentally.
Also do they really want to set the tone that “I expect you to intuit what I want and I won’t tell you directly”? Sounds like an awful place to work, right out of the gate.
Maybe? non-desperate folks might waste the company's time. Usually companies can only give out one offer at a time. If the non-desperate person takes a while to accept, someone else in their pipeline may get an offer and flake.
We had a candidate agree to a start date and then cancel 2 weeks before because they decided to stay at their job.
> “I expect you to intuit what I want and I won’t tell you directly”?
They do tell you directly? "Do this hard uncomfortable task, where the task is study leetcode, completing a coding project."
I have only once had this be a problem and in a way, it was my fault for not noting the ambiguity in the submission. Every other time I have dropped a comment saying "you could have intended A, but also could have intended B so this is when I go back to product and request more information."
I never deliberately snuck ambiguities into coding challenges. I've used it in oral sessions for senior and above devs though. "I'm a product manager who wants to add a new feature. I don't really know what's involved but I want you to implement it. Feel free to ask me a million questions and we'll talk it through!"
I wonder why the author thinks this is something unthinkable and unrealistic. Small teams with strict ownership and skilled engineers. If you join a company like that you will inevitably find yourself in that situation.
It's common for candidates who have an internal referral to skip all these steps, because the referral establishes a base line of credibility.
Anyway, in lieu of a coding interview, I opened up a couple of tickets (one a long term change request, another a real life problem we encountered just the day before). I provided them logs and answered questions and watched the interviewees move through both scenarios.
We capped the time at an hour. The focus was more on the interaction between them and myself—it actually used the "real stuff" they'd encounter.
I was amazed at the diversity of the responses and how differently people approached a problem.
These assignments are so much closer to a simulation of doing the actual job than the industry standard LC interview. The only thing closer is work trial which has the significant problem of requiring candidates to not currently have a job.
Yes, complain all you want about what companies SHOULDN'T do. But then what do you feel is a better approach?
From my experience, you ask 4 software engineers what a proper interview should look like and you'll get 4 different answers.
Having to spend time to do some work before an interview to prep (like a presentation about your experience or something) seems reasonable, but a company should be able to test technical skills adequately in the 5+ hours they spend interviewing candidates.
I feel like the take home assignment trend is part of the backlash against doing coding problems live in interviews and I agree with the article that it's just worse. It takes more of your time and also opens things up to candidates cheating even more than live coding over zoom does.
Just stick with the live coding and ask good questions.
- The candidate needs to do lots of work while the company gets away with little to none. - The candidate never knows what exactly the other side will be looking for, so the only way is to put in more work. Like when the assignment didn't ask for tests, they might still be expecting them. - The candidate usually gets no feedback about the work he did. More importantly he usually doesn't get a chance to defend his solution. It's either great or garbage. - While the company can learn something about the candidate, the candidate will learn nothing about the company by doing the assignment.
One asked for a 20+ page personality assessment (perhaps completing that _was_ the test?). The other asked for an interesting technical challenge which I was happy to do -- for fun -- and then followed up with a request to write a basic web service which was... unexpectedly simplistic for the level of role I was applying for.
I pointed out a number of my public, solely maintained repos on Github illustrating exactly the same sort of thing they were looking for and turned down the followup assignment, but we decided it wasn't a good mutual fit.
A few months before my last interview I had taken up Python by solving about fifty projecteuler.net problems. I’m not fluent, but capable, and since no one is hiring me to write code anyway, I put it on my resume.
At my interview with a CTO he said, I see you have Python on your resume? “Yep,” I answered, and said pretty much the above.
“Do you mind if I open a shared session to code in”
“Sure” (I haven’t coded in front of someone…ever?)
“Do you know what Fibonacci numbers are?”
“Yes”
“Can you write me a function to return the Nth Fibonacci?”
That’s clearly not a very hard task, and fortunately I was up to the task and got the job.
If you really want me as a developer, pay me. As an industry, we might even set a standard rate for these homework assignments. I suggest a rate which is enough for me to say it wasn’t a waste of my time.
As an anecdote, I was interviewing at a startup that sent me one of these homework assignments. I told them I would do it for $1000 now, or whenever I got around to it otherwise. I never got around to it.
I'm not in favor of take home "go write this thing" projects, but what you're suggesting isn't a viable answer at all.
From the company’s perspective, they are indeed getting value from the homework assignment: an opportunity to evaluate a potential hire. If that seems weird, consider why any company would pay to post a job opening. The price they are willing to pay has nothing to do with the marginal cost of displaying the ad.
Not if they don't ask. If you asked, you'd find many who are willing.
> You are not doing work for the company when you do some test to demonstrate your skill.
You literally are, it's just not work from which they are likely make a profit. That is not the candidate's problem. This doesn't make it unreasonable for the candidate to ask for compensation.
Agree, such or similar unrealistic interviews are loss for both sides, plus they are plain STUPID. I've performed ~400 interviews that most included coding in my career with consistency and standardized report (can compare apple to apple).
Every time I get those I request to do a live coding assessment, if they are not willing to lose even 1h of their time they were not serious about you.
If you decide your process really requires a solo assignment you can offer a gift card to prove you appreciate the candidate (and don't be stingy, you are already saving a lot of money by not having 1 of your engineers interview)
I would rather do this than some tired ass LeetCode problem.
If you are the one doing the supporting of your collaborators it's pretty much just like a coding challenge.
"your coworker needs to Inver binary tree but doesn't know how, can you help him? ... also the Internet just returns interview worthy crap when you try to search"
It's one thing where companies are applying this to 1000 applicants for an entry level position, but they also use this tactic when they reach out to you and tell you how excited they are about your background and resume.
In reality they didn't evaluate your background or resume and are just cold-calling you to fill some portion of their "funnel", then ignore all their excitement and reasons they reached out to you and have you jump through random hoops.
do you show them a quick'n'dirty solution that ignores edge cases but shows i'm a pragmatic and not going to overcomplicate things?
OR
do you show something fancy that you'd never actually do in an real codebase that shows off my depth of knowledge and where my ceiling is?
Candidate: writes code matching current style as much as possible to show that you can adapt
Reviewer: Rejected. Didn't even use autoformatter, which takes literally zero effort, so clearly does not follow best practices.
Candidate: autoformats
Reviewer: Rejected. Changed more code than was really needed (separate commit or not), clearly showing they didn't try to keep the change as small as possible. If this were a real PR, they would be making life more difficult to the reviewers making it harder to spot the parts that were actually modified.
Candidate: keeps existing code as-is, but writes own changes following the currently accepted conventions
Reviewer: Rejected. They clearly can't follow an existing style and prefer to be a snowflake and "leave their mark" in the codebase.
Candidate: does any of the options above, and adds comment explaining the reasoning, including mentioning the other possible options that they thought about and their tradeoffs
Reviewer: Rejected. Candidate thinks too hard about trivial stuff, showing lack of focus on what really matters.
---
Granted, being rejected for those reasons above could mean that you dodged a bullet.
But yeah, like you mentioned in a different response, a live interview (coding or not) helps reduce these kinds of uncertainties to some extent, but of course these live interviews have other trade-offs compared to take-home tests.
Debugging (and setting up the debugger) is a core skill every developer should have.
Literally my first internship. Had to migrate a small system from C to (modern) C++ in the span of 6 months. Most files (and I'm going to say in the ballpark of 85%) had last been touched in 2003.
I started my intership in 2020.
- Use probability as your advantage. Or put it another way, perseverance works. Do recognize that interviewing is a hit-and-miss game. Say with your preparation your chance of getting into any of your top 10 companies is 30% -- a pretty low number just for us to steelman argument. Let's say you can try each companies three times a year (most companies have cool-down period per department instead of per company). Then in 5 years, you can try 30 times. The probability of failing all of them is 0.3^30 ~ 10^-16. That is, practically, zero. Of course, this assumes that each attempt is independent of each other, and we can maximize such assumption by keeping learning the lessons of failures. By the way, this is also a reason why one can see so many so-so engineers in a big companies.
- Do not cram all of the leetcode questions. That's insane. Remember the quote from Invincibles: "when everyone is a superhero, nobody will be"? It applies to cramming too. Instead, use the time to study fundamentals. Pick up Kleinberg's Algorithm Design, Levitin's Introduction to the Design and Analysis of Algorithms, or Skiena's The Algorithm Design Manual. Note these books are not that academic. They all focus on the process of designing an algorithm from constraints and basic principles.
- Do not worry about Leetcode Hard. Yes they will be asked from time to time by an interviewer, but see my suggestion #1. If you really want to try Leetcode Hard, at least ignore the ones that require deep ad-hoc analysis or the ones that require an aha-moment. They are talent filters or memory filters or obedience filters, but they won't improve your engineering expertise. Then, why bother? And trust me, it is actually rare for you to get a Leetcode Hard question.
What are references for anyway...
The problem is that there is financial pressure to game the system, which means it's worth spending time and money on getting through the hoops but not on improving your work after that. Resume preparation, leet code grinding, even references are tainted when the reference is aware of the financial pressure (this person wasn't that great, but do I want to impoverish their family? or they'll scratch your back in exchange for you scratching theirs).
Of course, it’s important to pair that with other factors, like their past projects and how they collaborate. What do you think would be a better way to assess technical skills without falling into the "arithmetic quiz" trap?
You're being tested on your general intelligence, not because anybody cares about binary trees or reversing linked lists or whatever.
A lot of interviewers missed that memo. They are in fact very focused on a specific algorithm or design approach. It's quite tiresome.
Otherwise it's "well yes it's not great but it's better that not doing it"
Applicants underestimate the cost of a bad hire.
How is a coding challenge not problem solving?
Collaboration can be part of coding challenges.
How in the heck do you plan to measure growth in a hiring process?
Uh.. just about every company i've worked at in the last 20 years has had little to no documentation and the guy who last touched the code left already.
In all seriousness, we just need to get together and refuse to do these types of problems (starting with hiring managers and strong candidates).
All the time now that WFH is normal and collaborating with a team member is 10x harder and 10x slower.
This has literally been my job for a year. That is an actually important skill to have.
Lol, I'm pretty sure that's been my job description for the past 20 years.
What gives?
Most finish it, some don't and still get hired. It's a workalong in which we see how they approach the problem, how they discuss it with me, and what they do if they get stuck. We tell them to go ahead and google things, just tell us what they're looking for.
Over 10 years, we've got a pretty good record of hires that are smart, professional, and effective. And without specifically trying to hire for diversity, we've had a surprisingly diverse range of hires. We don't care about "culture fit", it's more like "work fit".
So I agree that the sort of code challenge where you send them away and see what they come up with is both unfair and a poor indicator. The real hiring test in an interview should be "can I work alongside this person?"
Is this a joke?
My answer to that question: I'm doing it right now, and have had to do it at least occasionally in almost every job I've ever had.
No coding challenges.
No take-home assignments.
What is the alternative?
Obviously you can't do that publicly because it would affect the company's image, so you still need to have leetcode/coding/takehome steps to keep appearances, even if they don't contribute to the final decision.
A later employer was in the hiring tech space, enabling this kind of take home exercises. Lots of time was invested into getting signals as the developer was doing it because everyone knew that's just as valuable, if not more, than the final solution. But many companies using these take-home exercises do it just to filter out candidates. To them, it's a proof-of-work system that helps regulate the intake of their hiring funnels.
1. Said challenges should reflect the work done on a day to day basis, not something completely disconnected from the company as a whole
Google's interview challenges should be very different from say, a local web development agency's interview challenges. I've seen too many companies where either the challenge was way too difficult/complex for the job on offer (build a complex modern app from scratch for a junior/graduate level role, or solve a bunch of leetcode tests and HackerRank puzzles for a WordPress agency job), or where it was far too simple (I remember at least one software engineering interview for a React focused role where the challenge could be solved without writing a single line of JavaScript).
2. The requirements and environment should match a realistic working day
When was the last time you had no access to the internet, no access to an offline IDE, no access to premade software, no one to help, a boss standing screaming at you over your shoulder and a camera tracking your every eye movement in case you're 'cheating'?
Probably never right? Unless your company environment is a dystopian hellhole, this is probably not your typical set of working conditions. Don't expect interviewees to work under them, they're not kids in school doing an exam.
3. And that said projects should be somewhat realistic in terms of scope
Most people aren't coding a new website/app from the ground up every day/week. Expecting someone to create that in a few hours feels incredibly unrealistic, and an incredibly poor method of judging someone's skills in the role.
"Thank you for sending that along. I would like to withdraw my application for this position."
Do it anyway.
And if they don’t want to do it, we don’t want them.
Right the fuck now. That's what happen when you get saddled with whatever monstrosity the last coder(s) who quit commited or had to maintain themselves.
And then you get the joys of trying to frankenstein some modern js "component" developed like only React exists on some vanilla / jquery front. Also a personal rant for those modern js script kiddies: forms action attributes exist and work.
Some candidates with many years of listed experience had never used Git. That was a little surprising but maybe they'd worked at a Perforce shop or something. Nevertheless, they knew in advance they'd be using it today, and that's not exactly some obscure job skill we were asking them to learn and then forgot. OK there: "I didn't use Git at my last job but I read up on it, and I might have some questions." Not OK: "What's a clone?"
Others didn't have an editor or IDE set up. Just how? I get not taking work home with you, but you've never once written a little program to balance your checkbook or make anagrams or something else random? And again, we'd told them ahead of time they'd need it today.
It's OK not to be a Python expert, but if we give you:
def add(a, b):
return 4
And I'll say it: we had plenty of people fail at the fizzbuzz step, often in astonishing ways. One memorable person couldn't grok the concept of factoring it out into a function like:
for i in range(1, 101):
print(fizzbuzz(i))
So no, I'm not giving up coding challenges, not until I see a higher passing rate among people who list years of experience on their resume. If you claim you've been writing Node apps for the last 10 years, by gosh, I want to see that you can at least write a working function in JavaScript.
I can't claim to be always be good at it, but this is what I strive for as well. I like talking to a candidate about something they are experienced or even expert at, and asking questions until I find the boundary of their ability. It can be very illuminating what happens when they hit it -- some get defensive, some get enthusiastic.
Defensiveness might just mean they're being interviewed and I failed to adequately put them at ease. But usually when we're neck-deep in some specific topic, both of us will kind of forget that it's an interview -- especially since the topic is more in their area of strength than mine.
Otherwise, defensiveness often means they never actually wanted to understand the problem they were working on, they just wanted to use it to get a degree or ship something by throwing it over the wall and forgetting about it. Even someone who is heartily sick of their PhD topic (as in, everyone who is over 1/3 of the way to getting one) will be relieved to discuss the ideas behind it, the reasons why it caught their interest in the first place, when they don't have to do the work of coming up with rigid results and writing them up.
Enthusiasm is usually a good sign, though even there I have to watch out for excessive enthusiasm where they care more about the problem itself than the benefits of solving the problem, and are likely to waste resources in unnecessary pursuits of perfection.
Oh, and finding the limits of someone's knowledge doesn't require a genius, which is fortunate since I am decidedly not one. 3-year olds can do it just by endlessly asking "why?" You'll probably need to be a little more sophisticated than that, which is good since you'll be able to evaluate their ability to explain things to you in the process.
I do like the return key, though.
I like live coding challenges, something like a ~2 hour pair programming session, ideally modifying an existing project. I invest as much time as each candidate, while we are both exploring whether we want to work together.
If I can’t do that, I ask to see their prior work.
Good programmers have usually written a lot of code. Having them walk through and explain it usually gives a good idea of what they can do and how they think.
Sometimes you meet a programmer who only works on proprietary code and can’t share it.
In that case I ask them to explain the design of something similar, and write a modestly sized example of a component of that system. Watching them in their own dev environment for 30 minutes usually tells you all you need to know.
I've hired plenty of people without having coding challenges or any form of live coding. I've been happy with all of those hires. A former co-worker did some take-home coding tasks, which they'd then talk to the candidate about during the interview. I feel like those hires where worse in may ways, they certainly didn't stay around as long. That may very well be completely unrelated obviously.
For years people have been complaining that exams aren't realistic, that some talented people just don't do well in an exam situation and we've mostly come to the consensus that this is correct and mistakenly filter out highly talented people. So why wouldn't we apply the same logic to hiring?
If you're hiring for a specialist position some coding exercise can absolutely be in order, but I can't think of any reason to have them for a junior position. So I wouldn't recommend dropping coding tasks completely, but I'd apply them more selectively, otherwise you risk missing a number of really good hires.
Part of it may also be that so many companies and interviewers are absolutely terrible at doing coding challenges, but do them anyway, because Google and Facebook do them.
One approach that I've encountered (with a YC company) is that the first interview was actually a code review. One of the founders asked me to review some SQL DDL, some backend API endpoints. The DDL was missing some indices that were needed for the queries. It was using an integer ID field. The API endpoints were missing input validation, error handling, etc.
I thought this was a GREAT way to start an interview that tested for depth of experience and platform/language knowledge.
This actually inspired me to build https://coderev.app because the tooling for this felt like it would be clumsy for both the interviewer and it was certainly for me as the interviewee.
But a lot of times, seeing a candidate's portfolio -- if they have one -- is probably even more insightful than any coding exercise. When I've been on the hiring side, one of my favorite things to do is to look through a candidate's GH and ask them questions about projects they've done, why they chose specific technologies, etc.
i had this approach work even in china where people tend to be very submissive and afraid to speak up. i expected them to have trouble in the interview, but most candidates told me they felt very at ease even though they had never worked with a foreigner before.