This is then done for every possible combination of approaches (homework questions, whiteboard interview, pair programming, fizzbuzz questions, algorithmic questions, work-history questions, career-progression questions, mini-project, magical incantations, and so on...)
It may sound cynical but at this point it feels like no one has any new ideas about it and the same content keeps getting recycled in the form of yet another article and yet another comments section.
Half of these articles are from company blogs which exist largely as just a marketing tool.
Since there is no actual new thought, everyone just cargo-cults the Google approach.
It's a gamble, no matter how in depth or complicated your hiring process is (from personality interviews to whiteboarding to take homes). Being interviewed and doing well is a skill set and it's very different from a full time work-place skill set.
All of these interviewing processes that get refined and more lengthy just create a new class of people who are good at being interviewed.
I don't know what the solution is, but making candidates jump through 3-4 interview rounds, memorize algorithms just for the sake of it, and go through 4 to 8 hour on sites probably isn't it. Even if it happens to give you a better-than-average employee, what are you sacrificing (as a business) in terms of man-hours, productivity, and focus?
Personally I'd like to see more companies institute a temp-to-hire process. Bring in new candidates for a two or three week contract, see how they fit in, pay for the their time (at a discount), and then go from there. If it doesn't work out, you get some work done for your company, and they get some relevant experience for their resume.
Writing the perfect resume and re-reading CTCI for the 10th time won't tell you how well the candidate will perform, but seeing them actually work might.
As to your point about marketing - I agree. Not one of these approaches are really novel or helpful to anyone but the company themselves, and is just a glorified "look at how we took a chance" tactic.
In the real world, people generally don't put in their notice until they've accepted an offer at another company.
It would be great if the only reason people could be let go were legitimate, but that's hardly the case.
If you devise an interview framework or a hiring profile, no matter how many traits you want to evaluate(as the article points out), they will lump those together and measure in some scale of how likely they are to hire the candidate(sometimes that is even just binary), if a candidate can emulate their desired profile he gets a high "score" and gets an offer, the bigger the financial incentives to do this, more people will game the system(prepare for specific interviews), think, if you can play the part for a couple of 1-hour interviews, you can get a 6-figure salary at a tech startup, that is a high ROI.
Most companies can't stop that, there isn't limitless budget for interviewing so the interview process has limited resources to evaluate people, which leads to false positives. There are also a lot of talented people that don't practice their interviewing skills enough and end up as false negatives.
It's human nature, people need to evaluate things on a single criterion and you can game that. The same thing happens with school, you can work your way through school material to maximize your grades without learning much.
Also, every person you are interviewing has the option to interview with other companies. With that in mind, you should always be interviewing even if you aren't actually looking to hire at that moment. Sometimes you might find a really good candidate but, most importantly, you will have options yourself.
Last, I really like your idea of a paid temp position to assess fit. This will benefit everyone tremendously.
Is it the high salaries? In that case, do you see such testing in law, or finance?
Is it the technical difficulty? In a previous life (read: up to a few months ago) I worked as a mechanical engineer, in a pretty technical field (numerical simulation of structures). Additionally to demanding reasonably deep knowledge in a few areas, the job came with the responsibility of making sure that your designs do not kill or injure people.
I had never had a technical test in an interview before I interviewed for my current IT position. Nor have I heard of colleagues having such a test. Is it because most engineers in Canada are licensed (myself included), and most jobs require you be licensed? If that's the case, it's pretty funny, because at least in my province, the test you have to pass for the license is entirely about ethics and professionalism, nothing technical.
Part of the code of ethics though, is that it's on you as a professional to ensure you have the necessary competence to do the work.
The barrier to entry for software engineer jobs is all at the interview. That's why it's so hard, tedious, and complicated.
Tricky brainteaser questions are popular in quant finance interviewing. Example[1]. There are also discussion forums dedicated to it. Example[2].
Some electrical engineering firms will ask you to analyze complicated circuits in the interview. Some even have "take home tests" where the candidates design a circuit according to the interviewer's specification.
[1] http://www.streetofwalls.com/finance-training-courses/quanti...
1. Pay a lot. Barring trading, you'd be hard pressed to find a job that routinely pays 200K+ to 25-year olds
2. Are, at least most of them, not fundamentally difficult for a generally smart person, despite whatever people convince themselves of. I am not talking of the so-called 10x engineers here, who you are unlikely to find through standard interview techniques.
3. Pride themselves on NOT being credentialist (at least on the surface). You can't wake up tomorrow wanting to be a lawyer, read some books, and get a good job offer in a couple months. It's possible for CS jobs.
If I see a bridge, I don't need to be an engineer to gauge if it is likely to support my weight. But if the bridge is invisible, I probably want significant assurance that no one is pulling a fast one here, so to speak.
In a field like mechanical engineering, I assume an employer can take for granted that a new grad has a certain set of skills (general physics, structures, fluids, etc.). That can't be assumed in quite the same way in software engineering.
Medicine, law, pharmacy, actuarial science, accounting...
For most upper class jobs it's the norm.
Make the process a conversation rather than an interview. Get to meet the person, have lunch with them, talk about what motivates them and what they're passionate about, discuss tech stuff, and at some point during that hour or two both you and them will have an idea whether you want to turn it into a steady relationship. :)
I like to add a code review stage to the mix, where we show them a small chunk of intentionally badly written code that might come up during a code review, which has both obvious and more obscure issues related to syntax, clarity, efficiency, etc., and you judge how they approach it and what issues they bring up. I've found this to be quite a useful metric that mostly replicates a real world scenario, but in a more comfortable setting than asking them to write code on a whiteboard.
And you try your hardest to block any biases for/against them you might have when making your decision.
By volunteering along side the company doing the software build(which was also volunteering, mostly) we were able to get the project on track, and update the client's expectations to a more realistic view of the timeframe and outcome.
By volunteering alongside them, for another org that we both were happy to support, I didn't feel like I was doing free work for them, and they weren't struggling to pay full salary and onboarding cost to find out the same.
Just a thought for an approach to get a sample of each other without one party doing free work for the other.
I also work at the company this article is about, and the article was written by one of my colleagues. While no process is perfect (including ours -- both the one detailed in this article, and the one we use for general engineers), one of the things that I like about working where I do is the emphasis on putting thought into the interview process and questioning a lot of things that get assumed without a second glance by interview processes elsewhere.
For an entry level engineer, this removes a bunch of variance and also because it gets you another scenario which happens way often in actual work.
You show up with the code and it gets reviewed & discussed on how/why of the implementation.
The most talented entry level people I've worked with have struggled with criticism instead of grading (i.e "grade the code" through a bunch of test-cases is different from "why did you do this and explain").
I recently had an old boss try to bring me on to a new job with him, but wasn't able to get me through the interview phase, despite knowing I'm capable, because my interviews with the rest of the team went so bad.
I've been a software engineer over ten years. All of the positions that I have been offered have come from especially kind/empathetic teams, and the interviews usually focused more on conversation than quizzing.
It requires an unfortunate amount of rejection before coming across my more selective criteria, but I guess that is what it is.
Problems: the existing employee ends with even more latitude to wreck the interview. You need to ensure that the interviewee is taking the lead. You probably also need to record the whole thing for objective review by someone else, which you'll have to disclose, which could make it awkward again.
Ed: I've also heard of processes where you put multiple interviewees on a task together. That has some similar potential benefits.
In most cases you are not looking for the people with the most level of CS knowledge, but rather someone that can think logically, that learns/ramps fast and have at least some explanation, even if wrong, for his choices and is able to take the observations/review as a learning opportunity.
While I can't be absolutely sure that they've done it themselves, a big part of the in person interview is based on the homework, asking them how they would extend it, add specific features, potential performance problems they see, etc.
Those answers are hard to fake if you didn't do it yourself
We had one case where a candidate used some open source code snippets in his solution, not declaring and removing the comments and license headers. Still it was very obvious in the difference of coding style.
I'm just underselling myself, shooting too low. At this point do I need to be applying for management?
1. Unemployment can be extremely stressful and there is a very good chance the person needs a job NOW. Every little thing you do to dilly-dally instead of making an offer is another hurdle that may send the applicant elsewhere. Act as soon as possible when you see a résumé you like (get on the phone, find out what you want to know, make a decision; if you bring them to interview and they’re struggling, give feedback immediately; don’t have a system where you wait 3 weeks to tell them yes/no).
2. Any effort you expect an applicant to spend will be greatly multiplied. That person is not talking only to you, they’re looking at LOTS of job postings!! If every potential employer starts assigning “homework”, it becomes a major hurdle and you can bet they’ll look a lot closer at an employer that might provide a paycheck in a couple weeks instead of “homework”.
3. You don’t need complex problems to understand if somebody can code. Let them use whatever language their résumé claims they know, give them something to do for a few minutes and see what happens. If they get stuck, give them a solid hint and again, see what happens. You’re trying to assess things like (a) if they appear to know what their résumé claims, (b) if they are the kind of person you can interact with to get somewhere on a problem.
What do we think the median interval from "established intent to interview" to "offer" is at medium-to-large software companies in SFBA? If I had to guess, it's on the order of a month. Nobody's hiring homework process puts a dent in that timeline. On the other hand, when done well, "homework" makes it easier for engineering teams to make confident decisions, which expedites offers.
How much effort do you think it takes a typical candidate to get through an interview gauntlet at a tech company with more than (say) 50 employees? It's at least multiple full days worth of aggregate effort, almost always including one actual, on-site full day. To the extent that homework offsets on-site interviews, it reduces the amount of effort candidates need to put in.
You basically already know what an entry-level hire will know: they’ll remember most of what the last year of their degree taught them, and have a fuzzy memory of anything earlier. They’ll be about as good at problem-solving as you can see on a whiteboard. You’re not paying ultra-senior salaries out of the gate; you’re just getting what you pay for so bring in somebody promising, start them on a project and let them begin a career with you.
Like I said, put yourself in the “new hire” position. They should be treated better given their likely circumstances:
You have to survive, you need rent and food and whatever else. You have debts. You’re probably working some non-ideal job (or 2 jobs) in the meantime for enough cash, enjoying all the no-free-time-left that gives you. If you’re really lucky, only a handful of companies have given you some stupid assignment to do with your spare hours, and since it’s “homework” and you have “more time”, they probably dreamed up a more involved problem than the average whiteboard question would be. Then you feel extra pressure because you really want to get it right but in reality you probably ended up rushing the thing and making some silly mistakes that you won’t see.
Never mind that “homework” is just the latest example of unnecessarily-long-winded hiring practices. Once you add up every little thing that every company wants, you’ve spent a lot of time that you don’t really have. (Some want you to create a new web account and profile that redundantly re-specifies 90% of your résumé. Some require you to specify your résumé as a file, then they scan it and scatter it into the wrong fields and discard the rest, requiring you to fix it all. Some have other forms for you to fill out. Some want multiple phone interview half-hours of your time, multiple on-sites, etc. and still can’t get back to you in under 3 weeks. Now do that 10 or 20 more times and don’t miss your next shift at the coffee shop.)
Who needs a full 1 day of interviews plus time for phone interviews for an entry level job.
I like the homework idea. I don't know about "coming back" but maybe send it to them a couple days prior to the tech interview.
I've personally been in the situation of receiving a list of 10-12 required features, with a stack that the recruiters know I am unfamiliar with, and then been told to only spend four hours on it.
While this was an extreme case, I find that the unreasonable time expectation is not uncommon. That is quite frustrating as an applicant.
with experienced people they always think they are at or near the top of the scale, but dont understand that you have a different scale entirely.
with a junior person, ask them to describe a project they worked on. if they 'i dunno, school stuff', then you're done.
if you can engage with them meaningfully about their school work, or if they've done personal projects, have developed any kind of insight, show a spark of excitement or creativity then you're done - hired.
just keep an eye on them for a while to make sure they're engaged and absorbing culture. invest a good amount of time in their success (mentorship, pair debugging, whiteboard talks). if they dont start contributing more than they are costing in time after a few months, find them another role or fire them.
Even the company that eventually hired me didn't look at my website.
edit: oh, maybe i wasn't clear. school projects are fine grist, you just have to be able to say something interesting about them
I would assume you'd be asking about how the address and data busses work which varies by cpu and architecture (von Neumann vs Harvard) presumably you'd be happy if some one explained how PIC memory access works ?
Well...
> Some of the qualities we looked for in our associates in particular were:
> Resilience: Learning on the job is hard and we assumed that the associates would make mistakes and struggle through difficult concepts. We needed people who could endure these struggles and bounce back ready for the next challenge.
> Willingness to learn and the initiative to do so: Clover would assist the Associates in their growth, and provide teachers and mentors to help along the way, but any incoming Associate would need to be responsible for their own growth.
> Humility: This is an important trait for all Clover engineers, but we paid special attention to it in our Associates. They would have to learn from those around them, be respectful of others, and be able to take difficult feedback with grace.
> ...
> Deciding the technical skills to evaluate was a long process. We expected to teach our Associates most of the technical skills they would need to do their job, but we couldn’t accept candidates that were completely blank slates.
It really sounds like this is measuring the candidate's "potential".
I would argue they defined "potential" as "resilience, willingness to learn, and humility", which they feel is an atypical definition.
(Personally, I think that's a decent set of criteria to look for in entry-level engineers.)
The task was super simple - basically just duplicating some of the configuration and renaming some values to create another job, but at the time I just froze and couldn't see it. I managed to get some sort of answer out, but wasn't happy with my response. They claimed that they were meant to give this to me in advance as a homework question and somewhere the ball got dropped, but I had a feeling they wanted it to be a more on the spot thing considering the problem was so simple.
Thankfully I had the wherewithal to ask if I could take the problem home with me after the interview. When I got home and looked at the problem with fresh eyes I facepalmed so hard. I wrote up the correct answer and emailed it over to the company recruiter, hoping she would pass it along to my interviewers, but didn't get my hopes up.
I guess she did, since I ended up getting the job. I still cringe a bit about how I just blanked on that stupid question. Thankfully I got ramped up really quickly once I actually started the job and had no problem figuring things out after that.
If our expectations are higher, then what I look for is code sensibility. Do they cut and paste the code without thinking "hey, this is kind of dumb to do"? Do they see repeated code and think "this can be collapsed into something more succinct"? This is the hardest thing to train for, and the most time consuming from a mentor point of view, so if we have a high bar for entry level programmers, then code sensibility is one of the factors on top of the 3 above that I look for.
I've rejected interns that worked for me from a permanent position because after 4 months, they still had terrible code sensibility. I had to pour through their code with a fine tooth comb because it was riddled with tiny bugs, and it needed constant rewrites because they just didn't improve over the course of the internship. I've had other interns that just "got it" and wrote great code from the start, or only needed a few reviews and then "got it". Those are the ones that will be productive quickly and won't hamper the rest of the team with lengthy reviews.
I am an experienced dev, but couldn't tell you my birthday in a technical interview. All the rest of the interview is fine, I only 'freeze' up at technical questions.
Sucks.
In theory, I like the approach they've taken to hiring associates, but at the end of the day, what really matters is: are the people they end up hiring successful at their jobs? It might take a year or more to really answer that question.
I don't feel like we're very good at this. As 'kinkrtyavimoodh points out in another comment, we see a lot of "how to interview engineer" type posts. But we never find out if, in the end, they were effective. This article claims that everything went really well because they had (or believed they had) such high-quality candidates that it was difficult to choose at the end, but they don't actually know that. They won't find this out after evaluating actual on-the-job performance. It might have been hard to pick from their candidate pool because their process failed to tease out flaws that are only going to become apparent later. We just don't know, and likely will never know.
If they asked the subjects to modify the homework slightly during the debriefing, that might be enough.
