I personally detest abstract puzzles like "why are manhole covers round?" or (worse) "How do you move Mount Fuji?" AFAIK these types of questions are not and have never been used in engineering hiring at Google (sales, etc I know nothing about). WSJ and other sources have made this (AFAIK inaccurate) claim. It's possible individual interviewers do this.
My own philosophy is that a simple whiteboard coding problem is an incredibly useful litmus test. I don't mean anything incredibly complicated either. If you can't write a function that gives me a row of Pascall's triangle, that's a problem. Note: I don't expect you to know what Pascall's Triangle is, merely implement it when explained.
Some of you may think such a test is a waste of time but I assure you it is not. It's astonishing how many people can't do this (and even more astonishing how many that do make it through resume and even phone screens).
The key here is "simple". One problem is that interviewers fall into the trap of thinking these problems are too simple and make the problems increasingly harder. Worse, they may get in a situation of asking someone a problem that is either something they know (because they've covered it before) or they don't (where it's not something you can simply nut out). This is a useless indicator in my experience.
Just because you can create a function to return a row of Pascal's triangle doesn't mean you're a great programmer but if you can't, it almost certainly means you aren't. It's a negative signal filter, nothing more, but an incredibly quick and useful one.
API pop quizzes I'm not a big fan of either, generally speaking, but on the other hand if you've used a language sufficiently long, basic questions about commonly used libraries aren't unreasonable either.
Interviewing is a numbers game. Your goal is to come up with a process that balances time spent by the interviewer with finding a suitable candidate. Note that I didn't say accurately assessing each and every candidate. It's sufficient for the employer to simply find one qualified candidate even if you falsely determine someone who is qualified isn't.
A certain error rate is to be expected. It reaches a point where the time investment to reduce it outweighs the benefit of higher accuracy.
One final note: I detest (both as an interviewer and an interviewee) any kind of "take home" test (other comments have mentioned this).
As an interviewer, it's extra work to assess, people cheat, good candidates won't bother and so on.
As an interviewee, I'm not going to spend hours on you without speaking to someone first to find out how likely it is that you're a good match for me and the likelihood that you believe I'm a good match for you.
I would argue that a take home test is significantly more effort than a simple coding problem that doesn't have a significant increase in accuracy.
The one exception to this is automated testing where the problems themselves are relatively interesting. Some of the Facebook Puzzles fell into this category (and some don't). People may do those anyway for kicks. But again, there's nothing stopping someone going to Github and copying and pasting someone else's solution so what value is the filter, really?
EDIT: clarified my Pascall's Triangle comment. This isn't a trivia problem. I don't expect people to know what it is, merely implement it when explained.
I have over 10 years of programming experience and I probably would have miserably failed your "litmus test", especially if it were in an interview situation in front of a whiteboard. Maybe this just means I'm a bad programmer, but I tend to think that problems like "Pascal's Triangle" don't represent what we work on day-to-day.
I was astonished at the 25 minutes you claimed this took you, so I did it myself. It took me about two minutes to write this, which worked the first time I tried it:
#!/usr/bin/env python
def pascal(i):
if i == 0:
return [1]
else:
L = pascal(i-1)
ret = [1]
for i in xrange(len(L)-1):
ret.append(L[i] + L[i+1])
ret.append(1)
return ret
if __name__ == '__main__':
import sys
print pascal(int(sys.argv[1]))
Am i a bad programmer now if i cant get a running program in less than 5 minutes ?
I would rather let people apply with code they are proud of like in every design job than solving litmus tests/puzzles
There are plenty of software jobs where that statement is absolutely true. I am not interviewing people for those jobs.
Taking a well-defined problem such as "write code to print the Nth row of pascal's triangle" is substantially easier than most of what we deal with on a day to day basis. The hard part is discovering and defining the problem well enough to get the point where it is as easy as "here is what we need to do, write some code to do it". Once you get there, whipping out the code needs to be second nature.
As cletus said in his post - a problem like this is a negative filter. If someone has a hard time whipping out the code for a well-defined problem like that, they will have a very hard time here.
If they can't, it doesn't mean they are a bad programmer. It doesn't mean they can't be successful in the industry.
It is simply a strong signal that they would not be successful at Google. And since I'm interviewing people for jobs at Google that's what I care about :-)
Would you rather be tested on your ability to comprehend a multi-kloc codebase and make correctness-preserving modifications to it?
That depends on how rigorous and well-maintained the existing test suite is, or even worse, that it is only half done. Which half? Guess, I think you'll be surprised.
You do not have to create perfect code on the whiteboard. You just need to demonstrate that you can code. Sounds like you would have been fine.
You are right! In fact I seriously doubt any serious software project that is solving business problems ever works on these sort of problems.
These problems are basically more of testing mathematical knowledge through programming.
They may make a great interview to hire professors but not programmers.
https://gist.github.com/1567842
I tend to like to rough code out Lispishly, but if I'm actually running stuff I'll often do it in C.
I've developed high-performance medical imaging applications. I've build large distributed financial systems. I've built 2D/3D games. I've done a ton of stuff. Hell, I'm a software engineering hiring manager at my company. Yet I have no idea what a Pascall's triangle is, or how to give you a row of them.
I consider questions like that to be as useless as brainteasers in determining whether or not to hire someone. Sure, it can help give clues about a candidate, but if you're seriously considering rejecting a candidate on that point alone, I think you've got a problem in your hiring process.
Pascal's triangle is simple a triangle which starts at 1 and where each number in the next row is the sum of the two numbers above it, with 0 being assumed for numbers on the edge.
A program to produce one is nothing more than a couple for-loops, printf, and some basic logic.
I would ask you to explain, you would smirk inside, I will be rejected.
I am sure google should have enough data by now to see if there is correlation between knowing this problem and performance of hired programmer. I will be suprized if it has any relation.
I'm sure cletus or any other interviewer would explain what Pascal's Triangle is to you, and provide a couple rows of example output if you asked.
EDIT: Find/replace "Pascal's Triangle" with "FizzBuzz" and reread the comment. You'll get the same insight without getting hung up on seeing a term you didn't know.
I have to ask the parent poster if they truly feel that people need to know this http://en.wikipedia.org/wiki/Pascal%27s_triangle to deliver web or mobile solutions? You are highlighting the issue that the article is talking about.
Further, a whiteboard is like giving a mechanic a drawing of a car and a hammer and saying fix the head gasket. If creating a row from a Pascal Triangle was a real issue that crossed my desk the first thing I would do is hit Google to find the mathematical formula and explanation of what it does then I would see if there is a standard library and finally I would write it if one does not exist. Your whiteboard does not have search capabilities on it, so how are you expecting someone to show you how they work when you have provided them with any semblance of an environment that they would work in.
I don't think the point is that you would be expected to know every historical abstract concept, but rather that given the basics of one, you could write code to implement a data structure around it.
If someone genuinely has never seen a problem before, and then figures it out in 45 mins, this person may look worse than someone who memorized the answer, and solves it in 10 mins.
The best solution is to give a previously unknown question to both the interviewee and the interviewer, and have them work out the solution together. It doesn't have to be a coding question, but the interviewer should be good enough to assess whether or not the person they are interviewing is good, not from how quickly they spit out an answer, but through how they work to get the answer.
1) I don't program on a whiteboard. It's 2012, I have tools, and they beat the whiteboard setup every time.
2) I will never implement something that I have not implemented before without at least first consulting with a search engine. You see, I am not a genius that will come up with the best algorithm for a calculation every time I am put up to the challenge. My amateur attempts will be no where near as good as what others have collectively accomplished over the years that the problem existed.
For example, in this particular case (I have never had to implement a function that generates Pascal's triangle before), I can guarantee you that anything I came up with at a whiteboard right now would be completely inferior to what the result would be if I was allowed to do a search first. So unless you're trying to see if I am capable of writing any old crappy code so long as the result is correct, your test isn't adequate.
3) Interviews still make me nervous, doesn't matter how many times I have them, I never feel comfortable going through the process. This often leads me to simply draw blanks when given a whiteboard assignment of this nature. You may think this means I am just incapable of solving your problem, or that I perform poorly under pressure, but I can tell you that this type of "put you on the spot" pressure is much different from the type one generally feels in a work environment.
2) Nobody cares how good your solution is, they care about seeing your thought process while solving it. They are likely waiting breathlessly to ask you follow up questions about your poor implementation that will further enlighten them about the way you think. It would be a downright shame to disappoint them by writing a great solution right away.
3) This is a legitimately big problem. Everyone responds to the pressure of interviews differently and it is definitely not necessarily correlated to their response to the pressures of the workplace. Ideally you would notify the interviewer(s) of your anxiety beforehand and they would be willing to work with you on either a short take-home (which a lot of people also hate for other reasons), or some type of working interview (which is often infeasible).
The original article was about a specific-language whiteboard problem, which is open to arguments about tooling and internet access, but many people ask these sorts of questions in terms of pseudocode, structured written english, or even discourse out-loud, and the goal is not to determine programming language or algorithm knowledge, but problem solving skills.
edit: phrasing
It's almost like Google has some sort of informal process for throwing away old interview questions this way, just for laughs...
This also makes me think I may spend too much time lurking on HN.
Well, I can assure you that as a kernel programmer, I think it IS indeed a waste of time. It's also astonishing that the moment the interviewer asks me something like this, my enthusiasm and interest in the interview drops by more than 50%. When I apply for a kernel programming a job at Google, I am dying to talk about my system internals knowledge, device driver programming experience, subtle differences between the AMD/Intel arch and what not. Instead, what do I get ? The recruiter asks me to recite by heart problems from a popular algorithms book for the pre-screening. It's like the prospective employer is telling me : I don't care what you did in the last three years. Let's start from scratch!
What also bothers me is the the test for coding skills. When a company hires a Linux kernel developer, the best possible way to test for coding skills is to look at my contributions. Instead, I am asked to write code and implement malloc so that the interviewer gets a taste of my coding knowledge. I think this is a huge waste of time.
My point being that the job interview at Google is incredibly generic. I believe that just like you should have a different resume for each job you apply to, the same applies to the interview process too.
More Info: This is very specific to my interview experience at Google. The two times that I applied at Google for Linux kernel development positions, overall I had the same experience as described above.
Some more Info : I also don't understand the idea of the algorithms test during the prescreen. If you go through the algorithms book the recruiter suggests and look up on glassdoor for interview questions, there's a very fair chance that you already know the answer to the question that's being asked!
Most of the supposed brilliant Algorithm and Math brains are basically people who crawl internet forums for puzzle questions by spending around 30 minutes everyday.
They don't know a jack about algorithms or math. Its just they know enough to game the interview.
-module(pascal).
-export([tri/1]).
tri(Depth) ->
tri(Depth,[[1]]).
calc_row_rest([_]) ->
[1];
calc_row_rest([X,Y | Rest]) ->
[X + Y | calc_row_rest([Y | Rest])].
calc_row(Acc) ->
[1 | calc_row_rest(Acc)].
tri(X,_) when X =< 0 ->
[];
tri(1,Acc) ->
lists:reverse(Acc);
tri(Depth,Acc) ->
NewRow = calc_row(hd(Acc)),
tri(Depth-1,[NewRow | Acc]).
> pascal:tri(10).
[[1],
[1,1],
[1,2,1],
[1,3,3,1],
[1,4,6,4,1],
[1,5,10,10,5,1],
[1,6,15,20,15,6,1],
[1,7,21,35,35,21,7,1],
[1,8,28,56,70,56,28,8,1],
[1,9,36,84,126,126,84,36,9,1]]1. It falls on the interviewer to be helpful and clear enough to help a candidate understand what is being asked. Often, a brainteaser of any sort explained poorly can be a death knell for anybody, no matter how smart they are. It's imperative that the interviewer is able to thoughtfully analyze a person's line of thinking moreso than whether or not they can get to a right answer.
2. Which brings me to the second point... The issue with a lot of brainteasers for poor interviewers is that it's a binary outcome. Either the candidate already knows the answer and comes off looking like a genius, or the candidate stumbles around in no man's land with no clue what you're talking about. Again, it's key here that the interviewer is able to guide a candidate through the thinking instead of sitting back and enjoying the show (which is admittedly sometimes fun).
I definitely think there is space for brainteasers, but it really requires an interviewer with the right training to get it right (which a company like Google does not have to worry about, but a start-up may). What I've actually heard is an interesting approach for start-ups recruiting is to just have the candidate sit and try to tackle a recent problem the company has faced with the interviewer.
We did not actually give people tests to take home. :)
Interviewing is a numbers game. Your goal is to come up with a process that balances time spent by the interviewer with finding a suitable candidate.
You have to admit that this is biased towards a company, like Google, that must hire many more people per unit time than most companies do. (And the Google process seems to work remarkably well for Google.)
A more useful example was when I asked my high school AP Computer Science class years ago to test if an array of integers contained a poker straight without using a parser, assuming ace=1, jack=11, queen=12, king=13. The key was to see if they knew how to think about algorithms. I wanted them to perform a sort and iterate across the array, but was interested in what they came up with. It tested a lot of important concepts in thinking about problems, which is more important to me than correct syntax on a whiteboard or implementing valid recursion on the fly.
I watch them work and help them out as they go; I'd get a ton less info if I just looked at the finished product.
We do also have a couple of "off line" coding tasks, which are even more informative (if I had to ditch one, I'd ditch the live one).
import scala.collection.mutable.HashMap
import scala.math.BigInt
val pascals = HashMap[Int,List[BigInt]]()
def pascal(row:Int):List[BigInt] = {
val result:List[BigInt] = row match {
case 1 =>List(1)
case 2 =>List(1,1)
case n =>List.tabulate[BigInt](n)(x=>{
if( x == 0 || x == n-1 ) 1
else pascals(n-1)(x-1)+pascals(n-1)(x)
}
pascals+=(row->result)
result
}
(1 to 100).foreach(row=>println(pascal(row)))
------
Results: ---- List(1)
List(1, 1)
List(1, 2, 1)
List(1, 3, 3, 1)
List(1, 4, 6, 4, 1)
List(1, 5, 10, 10, 5, 1)
List(1, 6, 15, 20, 15, 6, 1)
List(1, 7, 21, 35, 35, 21, 7, 1)
List(1, 8, 28, 56, 70, 56, 28, 8, 1)
List(1, 9, 36, 84, 126, 126, 84, 36, 9, 1)
List(1, 10, 45, 120, 210, 252, 210, 120, 45, 10, 1)
...and 90 more rows -------
Prints Pascal triangles some 10,000 rows & beyond. So where's my Google job ?:)
Kidding aside, In 1999,I got hired at Sun Microsystems based on a 1-hour written exam. Chap gave me a pencil, paper & few programming problems & left me alone in a room. No laptop, no books, nothing. After 1 hour, he grabbed my paper, looked through the work & said "You are hired". Then we went for coffee.
otoh, I didn't get hired at 2 quant firms in 2011 based on this Whiteboard style interviewing. I got the first puzzle right & then halfway through the 2nd puzzle the chap said "There's an easier solution" & then they upped the ante ( Stirling numbers & then convolution integral & then I think they decided I wasn't a good fit because I couldn't work out every single theorem right there and then. I can do any number of problems, but if you could please write them down on a piece of paper & leave me alone for an hour...I can't quite think and write in front of you while you are passing snide comments on my thought process. Most of my professors when faced with a problem they haven't seen before typically say "I'll get back to you", and then in an hour or two, they call me to their office and show me their work. They don't try to work out a solution right in front of a classroom of 100 gawking students. They do so in the privacy of their office. No offence ( cause I still want that google job :-)
def print_pascal(rows)
row=[1, 0]
(1..rows).each do |r|
next_row = Array.new(r + 1)
next_row[0] = 1
(1..row.length-1).each do |i|
next_row[i] = row[i] + row[i-1]
end
print row[0..-2], "\n"
row = next_row << 0
end
end
print_pascal(100)in ruby
def fact(n)
n == 0 ? 1 : n.downto(1).inject(:*)
end
def pascal(n)
0.upto(n) {|r| print "#{fact(n)/(fact(r)*fact(n - r))} "}
endNot that anyone will ever read this... I thought I'd managed to escape this time, but then I thought it might be some good practice, so I timed myself...
nitrogen@n2:~ $ time bash
nitrogen@n2:~ $ vim /tmp/pascal.c
nitrogen@n2:~ $ cd /tmp
nitrogen@n2:/tmp $ gcc -std=c99 -Wall -Wextra pascal.c -o pascal
pascal.c: In function ‘main’:
pascal.c:15:21: error: expected ‘;’ before ‘)’ token
pascal.c:15:21: error: expected statement before ‘)’ token
pascal.c:16:2: error: too few arguments to function ‘calloc’
nitrogen@n2:/tmp $ vim pascal.c
nitrogen@n2:/tmp $ vim pascal.c
nitrogen@n2:/tmp $ ./pas^C
nitrogen@n2:/tmp $ gcc -std=c99 -Wall -Wextra pascal.c -o pascal
nitrogen@n2:/tmp $ ./pascal 1
1
nitrogen@n2:/tmp $ ./pascal 2
1 1
nitrogen@n2:/tmp $ ./pascal 3
1 2 1
nitrogen@n2:/tmp $ ./pascal 4
1 3 3 1
nitrogen@n2:/tmp $ ./pascal 5
1 4 6 4 1
nitrogen@n2:/tmp $ ./pascal 6
1 5 10 10 5 1
nitrogen@n2:/tmp $ ./pascal 7
1 6 15 20 15 6 1
nitrogen@n2:/tmp $ exit
exit
real 4m5.621s
user 0m0.928s
sys 0m0.136s
#include <stdio.h>
#include <stdlib.h>
int main(int argc, char *argv[])
{
int row;
int *data;
int i, j;
if(argc != 2) {
printf("Usage: %s <row>\n", argv[0]);
return -1;
}
row = atoi(argv[1]);
data = calloc(row, sizeof(int));
data[0] = 1;
for(i = 1; i < row; i++) {
for(j = i; j > 0; j--) {
data[j] = data[j] + data[j - 1];
}
}
for(i = 0; i < row; i++) {
printf("%d ", data[i]);
}
printf("\n");
return 0;
}public class Pascal
{
public static long[] getRow(int rowNum)
{
long[] thisRow = new long[rowNum];
if(rowNum == 1)
{
thisRow[0] = 1;
}
else
{
long[] lastRow = getRow(rowNum - 1);
thisRow[0] = 1;
thisRow[rowNum - 1] = 1;
for(int i = 1; i < rowNum - 1; i++)
{
thisRow[i] = lastRow[i - 1] + lastRow[i];
}
}
return thisRow;
}
public static void main(String args[])
{
int getRowNum = Integer.parseInt(args[0]);
long[] pasc = Pascal.getRow(getRowNum);
for(int i = 0; i < pasc.length; i++)
{
System.out.print(pasc[i] + " ");
}
System.out.print("\n");
}
Edit, cleaned up code:
How do I paste into here whilst preserving line breaks?
function printPascal(x, triangle) {
var row = [];
while (x) {
var next = clone(row);
next.push(1);
for (var i = 0; i < row.length - 1; i++)
next[i + 1] = row[i] + row[i + 1];
row = next;
x--;
if (triangle)
print(whitespace(x) + row);
}
if (!triangle)
print(row);
}
function clone(list) {
var clone = [];
for (i in list)
clone.push(list[i]);
return clone;
}
function whitespace(x) {
var white = " ";
while (x) {
white += " ";
x--;
}
return white;
}
printPascal(10);
pascal = iterate (\xs -> zipWith (+) ([0] ++ xs) (xs ++ [0])) [1]
void pascal(int n, int *buf)
{
for (int row = 0; row < n; row++) {
for (int prev = 0, col = 0; col < row; col++) {
int curr = buf[col];
buf[col] += prev;
prev = curr;
}
buf[row] = 1;
}
}Looking at the C version, I wondered if it could be done without temporaries curr and prev. My no-temporaries version is not really any nicer, though it does shave off a few lines. :P
void pascal(int n, int *buf)
{
for (int i = 0; i < n; i++) {
buf[i] = 1;
for (int j = i - 1; j > 0; j--)
buf[j] += buf[j - 1];
}
} let rec iterate f value = seq {
yield value
yield! iterate f (f value) }
let pascal = iterate (fun xs -> List.map2 (+) (0::xs) (xs @ [0])) [1] gimli> pascal <- local {
gimli+ loop <- function(n, xs) {
gimli+ if n > 0 then do
gimli+ print(xs);
gimli+ loop(n - 1, [0, xs] + [xs, 0]);
gimli+ end
gimli+ };
gimli+ function(n) loop(n, [1])
gimli+ }
func(n) loop(n,[1])
gimli> pascal(0)
F
gimli> pascal(1)
1
F
gimli> pascal(9)
1
[1,1]
[1,2,1]
[1,3,3,1]
[1,4,6,4,1]
[1,5,10,10,5,1]
[1,6,15,20,15,6,1]
[1,7,21,35,35,21,7,1]
[1,8,28,56,70,56,28,8,1]
F
gimli> xs <- [1]
1
gimli> xs <- [0, xs] + [xs, 0]
[1,1]
gimli> xs <- [0, xs] + [xs, 0]
[1,2,1]I put something out in Java in about 20 minutes and 50 lines using a PT class, a newRow() method that swaps the current row with the previous row and sets up the new current off of the previous (prev[element] + prev[element -1]), and a printRow() method.
My solution works well, but I'm curious how you would assess this solution during an interview. Would my language choice be a pro/con? What is the avg time you see it take an interviewee to complete this problem and how many LOC is it usually accomplished in under the pressure of an interview?
pascal 1 = [1]
pascal i = zipWith (+) ([0] ++ pascal (i - 1)) (pascal (i - 1) ++ [0])
The above solution expresses the recursive solution as the sum of two earlier rows after appending some zeros. There is very little room for errors to hide and it provides you with a kind of 'definitional' starting point to begin optimizing if you want.
My implementation in 15 min using javascript. BTW I'm a freshmen CS major.
function row(len){
if (len==1){
return [1]
}else{
var p = row(len-1),
arr = [];
for (var i=0; i<len; i++){
if (i==0){
arr.push(p[0]);
} else if (i == len-1){
arr.push(p[i-1]);
} else {
arr.push(p[i]+p[i-1]);
}
}
return arr;
}
}Or as we say in the medical field, this test has high "sensitivity" :)
http://en.wikipedia.org/wiki/Sensitivity_and_specificity#Sen...
Now I've seen them I'll keep them in mind if I ever have a need for them.
Although, in my research it was stated that these were extensions to the ECMA-262 standard and may not work everywhere. Therefore you would have to write up your own Array.prototype.map and Array.prototype.reduce to make them work. Is that true? If so, wouldn't seem simple to me unless you use them on a daily basis.
(fn [r]
(reduce #(cons
(* (first %1)
(/ (- r %2) %2))
%1) [1] (range 1 r)))# print a row of pascal's triangle
print "1\n";
:)
- Good puzzles are actually a talent attractor; many smart people found out about our company via our puzzles.
- Good puzzles are ones that scale well; i.e., where the basic problem is pretty easy and can be solved by a decent programmer in a few hours, but with harder variants that can take much longer.
- "Take-home" puzzles (as opposed to in-person whiteboard tests) weed people out who don't really like to program and/or can't finish things; this is a useful filter. If someone complains that "they shouldn't have to spend three hours writing code to get an interview," that itself is a pretty good counter-indicator. (Yes, we are all busy. But you're talking about starting a relationship with the company that may last 10+ years. You can do three hours of prep work for your interview. And you have to code a fun puzzle in the language of your choice, not some subroutine in a 30-year-old COBOL banking system.)
- It seems that in-person whiteboard or locked-in-a-room tests are pretty poor indicators of success. Many good programmers put in this situation significantly underperform their true abilities.
- It's not a great idea to evaluate someone purely on the basis of puzzle-solving ability.
- Many one-liner puzzles are bad indicators, because you either need to "know the trick" or have memorized the answer. Many, but not all, Microsoft and Google interview questions I hear about -- I've never interviewed at either place myself -- sound like they fall into this category. (E.g., from a recent article I read about Google: "You're reduced to the size of a nickel and thrown into a blender. How do you get out?" I don't care if you're clever enough to answer that; I just want you to able to write enormous amounts of high-quality code for me.)
Im my experience, the best way to find out if someone is a good programmer is to talk to him/her at length about something he/she has built. Can he/she talk in detail about how it worked? What challenges were involved? What tradeoffs were made? And, above all, was there passion behind the work and these choices?
A good interview for me was one where the candidate came in having written or contributed to a large system -- for fun or work -- and was excited to tell me about it. It didn't really matter whether it was relevant to the work we did (airfare search). Anecdotally, it seemed like people who really loved to code generally worked out pretty well. People with great-looking resumes that didn't love to code -- but maybe loved other things, like arguing over which language, operating system, architecture, or business strategy was better -- usually didn't produce much.
If the first thing you want to do when you wake up in the morning is code something, you're probably going to do well as a coder on a hard-core software team. Otherwise, not. Everything I personally did on programmer-hiring strategy was a proxy for figuring out whether someone was like that or not.
But compared to coding and math, puzzles feel sterile. I don't get to build anything, and I rarely get the kind of insights I would get from figuring out a proof. (Raymond Smullyan's puzzles are a marvelous exception to this.) Even cryptanalysis is more fun, because there's a real opponent.
I know lots of amazing programmers who love puzzles, and that's cool. But if you only hired puzzle lovers, you'd miss out on a lot of good people.
Everyone likes to use Google as the example of the elite programming company. I've read the books about them, used the products, followed their progress over the years. So, sure, a part of their big success is hiring the 'smartest' college grads.
However, I've also worked with Ph.D.'s and even interviewed them for positions lower than ones I've held (I have not yet finished college - some day...). And I can tell you that book smarts != execution smarts.
I feel the same way about puzzles. I have never used puzzles or asked 'small' one off programming questions. Instead I've always been interested more in grep'ing the concepts and if the personality is the proper fit. In my mind, anyone can open up a book and read about specifics about O(log n). But how does a person persist at solving a problem and how do they (can they?) change direction if their solution sucks?
I feel like I value creativity over anything and they can creatively solve a puzzle, cool. But if they can creatively solve a task on the system I'm building and are a great fit to the team, then better.
I'm an API design geek. I've never seen a cool puzzle about API design, or something remotely related.
Lattice animals include polyominoes, which are probably familiar to most HN readers, but subsume other graph-related objects as well. I didn't know much about these, and ended up reading a bunch of cool papers on them as part of working on the puzzle. So at least for me, good puzzles have catalyzed some learning.
My mathematical preference is continuous math; my preference for using mathematics is to apply them into real problems.
I'm not saying that all puzzles are pointless, but I think most interview puzzles are kind of an empty exercise that only a certain type of person likes, and that only people who know the trick will get.
Of course, we couldn't easily do A/B experiments, so we'll never really know how the true attract:alienate ratio.
Trying to look at it objectively, I would probably be somewhat put off by being required to do a puzzle to "prove myself", but would then find the specific puzzles offered to be pretty interesting, and worth playing around with. Perhaps it's not tantamount to a $300 tax on ones life it is seemed fun and valuable in its own right?
It is much more like having to travel 90 minutes each way to get to where they want you to interview. That could be very offputting if you do enough interviews that three hours each adds up to a big number, but at least you don't need to worry about them being tempted to rip you off.
During the same job search, I also noticed Facebook had puzzles. I solved one and got invited to interview but never went. I wouldn't have applied otherwise. I guess this is another lesson - if you put public puzzles out there, expect people to send solutions without necessarily being excited to work at your company.
I'd never seen it before the first interview. In that first interview, I broke it down quickly into its component parts, and figured out where the trick had to be -- and from there quickly solved it on the spot.
In the second interview I spent a couple minutes trying to remember the solution from the first interview, and completely flubbed it. It wasn't the first part of that second interview that I flubbed, either, including arguing with one of the interviewers and basically telling him he was wrong.
Oddly enough, I got the job with the second company and worked there for almost six years. For the record, I love doing those kinds of puzzles. But I do video game development, so maybe puzzles are a bit more relevant to my industry (in that 80%+ of what we're doing is figuring out how to solve puzzles in the form of how to do something with less CPU or why a particular video card is behaving in a certain way).
I don't think you give yourself enough credit in the success of the puzzles. They were really well-conceived problems - just difficult enough without requiring too much domain knowledge. Very 'catchy', if that makes sense.
If I were asked such a question in an interview, my response would be: "Dunno. Let's talk about some real-world problems you're facing right now instead."
I dislike trivial questions, especially during an interview. I'm a professional, not a circus animal.
Bingo.
To me, this is the kicker statement for the industry. Solving fake problems like nickel-in-a-blender is for the clever. But I sure as hell don't want clever code solutions; I want rock-solid, reliable, fundamental solutions.
Anyone who feels these questions are an indicator of programmatic success is measuring the wrong things.
I sit on an engineering hiring committee and I can't really think of the last time I saw one of the ridiculous puzzle questions show up in an engineering interview. The WSJ article was pretty ridiculous.
i have done your puzzles for fun, but i would still be annoyed at having to do half-baked puzzles if i were suddenly out of a job and applying to a bunch of places.
(ps. and thanks for them :o)
We also never put a puzzle out to the world until a bunch of us had solved it. There was a long gestation period, and there's a "puzzles-discuss" mailing list for working out the kinks (and, in many cases, rejecting certain puzzles entirely). So, yes, you have to make a real commitment to do it well.
On the other hand, developing the puzzles was a lot easier than most of the coding on our real system, so it's all relative. It was actually really fun to do this stuff, and a nice change of pace. One of ITA's best developers (Justin Boyan) would take it upon himself to create the "perl postcard" solution for each problem. Another (Jim Rees) would always make an incredibly fast C++ version; it usually ended up being not only the fastest of ours, but faster than all the submitted solutions as well. I learned things just by reading his puzzle solutions. :)
I found the process rewarding beyond the hiring benefits, and I'm sure I'm not the only one. (Well, I hope so, at least.)
So, here is my opinion as a non-halfway-decent programmer.
I thought puzzle was fairly fun, but I like programming for two reasons:
* Building useful knowledge. Learning a new language, library, or algorithm I want to play with, programming is the natural way to learn.
* Building cool stuff that people like. Having an awesome product when I'm done with it gives the feeling that I've done a good job and produced something of value.
Programming puzzles don't really do either of these things for me. My motivation for spending three hours doing a puzzle is that I would like to talk to a person. I am basically assured that the puzzle will be used as a filter for the company to avoid talking to people. It will run against a test suite and if it fails it will not be looked at by anybody. This seems very impersonal. Is this the experience I should expect from the rest of the company if I'm hired?
I've got a Git-hub account with some of my code in it. I've spent time on it and enjoyed writing all of that code. I published it to the internet so I'm (at very least) not ashamed of it. Why can't you look at that to determine if I can program? Why do I have to write code specifically for your particular company?
This isn't an attack on ITA, it seems to work for them. But they are a unique case of a company where it seemed worth it for me to spend the time to do the puzzle. Any other company I would have just skipped it unless I genuinely had nothing better or more interesting to do (of which I almost always do).
When I applied at ITA they didn't even acknowledge the program I had written to go along with the application. Not even a quick 'it ran' or 'had the right output but we hired somebody else'.
I'm glad I didn't get the job because not having enough time to evaluate or even run a programming test you gave is also a pretty good counter-indicator of a good company to work for. And when I wrote back to express this the guy sheepishly admitted they had so many exceptional applicants so they just went with an Ivy-league graduate. Wow.
If you are going to require a programming test, be prepared to evaluate it.
I agree. It is possible the reviewer didn't think your code would run, couldn't get it to compile, etc. But, nevertheless, you should have received a fuller response than that. If I were still COO at ITA, I would ask someone to look into your particular case to understand what happened.
I worked on that one for a looong time, and what bugged me was the screenshot of the expected interface had a result returned in like 0.05675ms or something, and I couldn't figure out how to get it that fast. It took be forever to convince myself that it was just a mock and probably should have read 0.05675s (seconds instead of milliseconds).
We often had internal solutions (and externally-submitted solutions as well) that were orders of magnitude faster than the average. There was a very wide range of execution times. E.g., we had solutions to the Queens & Knights puzzle (in the archive) that ran in tens of milliseconds, if I recall correctly. Typical solutions we received took orders of magnitude longer.
Usually when one solution was a million times faster than another, it was because the fast solution exploited a clever algorithm or data representation -- e.g., representing an 8x8 chess board mask as a single 64-bit register rather than as an array of bytes.
(The sub-millisecond time was for real)
Many of the articles about Google interview questions on sites like Business Insider, at least for software engineering positions, are total bullshit. http://www.technologywoman.com/2010/05/17/debunking-the-goog...
But the programming puzzles on the website were just awesome, even though I just solved them because they were fun, not for applying for a job. I've been surprised that nobody else has done the same thing, but your explanation of how much work went into the puzzles kind of explains that :-)
I did one of them just for fun 3-4 years ago: "implement a scalable multi-user chat server without using a framework like Netty/ACE/EventMachine". Unfortunately, at the time I couldn't move to Boston, so I never submitted my solution.
Likewise there were puzzles that would appeal to web developers (e.g., build a typeahead web applications), machine learning/AI folks/etc...
But the puzzle process, as originally conceived, (in 2000!) was not necessarily supposed to be a hard filter. As an applicant, you had several ways to stand out, and doing a great job on a puzzle was one. Having someone who already worked for us say you were awesome was another. Being an open source superhero was another. Etc.
In actual implementation, though, it probably was/is used as a hard filter.
That's a very good point that I've not seen made so well before.
After so many failed Google interviews, I found it easier to build a company and sell it to them, instead.
I would argue that they don't and I believe that it is due to their hiring process. Now I do believe that they have very talented individuals but there organization reflects their interviewing bias as such so does their product portfolio. There is a reason that they have to acquire new products for their portfolio and that reason is that they are not recruiting the kind of people that generate new products. As such their hiring process should be suspect.
37signals hires for business value because, as a small company, it has to. Google is a huge company and doesn't have the same incentives. Google's size can blind it to the fact that its hiring practices optimize for mathematical knowledge and intuition over business value. And, as a business, Google is wrong and 37signals is right.
I don't believe that's the case. What I've seen with talent acquisitions (at Google and elsewhere) is that interviewing the team is actually prerequisite to making the acquisition. Of course, the interview doesn't have to be as rigorous: during the due-diligence process you can actually view the codebase and see the individuals' contributions.
What I've also seen, specifically in the case of Google, is that individuals from acquisitions would be asked to stay a year, at the end of which they'd need to interview for other positions at Google: if they couldn't pass the interview, they'd be let go with a very generous severance package.
That said, I am sure there are exceptions.
(Also, I've never heard of G+ being in any way a product of a talent acquisition.)
Generally whenever I've done this to recruiters their interest level goes up and they tend to be far more willing to listen to your concerns about the interview and go to bat for you. And recruiters tend to have a lot more ability than you might think in getting parts of the technical interviews largely ignored by hiring managers - including at Google.
Incidentally my impression from having talked to several Google recruiters is that a decent number of them are incredibly frustrated with the way the process work, and will work hard to get you past this hurdle if you want them to and provide them with enough ammunition (e.g. explain why the question was inappropriate for the position).
But when I'm interviewing, I do ask people for a pointer to their open source projects. And if they don't have open source projects, I give them a workstation and ask them to write code. I've seen too many self-proclaimed programmers who claim, "I spent the last 2 years writing Python," who can't sum a list of 10 numbers without using Google. (Hiring can break my heart.)
One way or another, you've got to see code. Real code is best, but it's better to ask people about toy data structures than it is to hire 3 team members who can't write FizzBuzz.
I did a hiring round once for my old team when I worked at Yahoo years ago, where out of 10 or so CV's the recruiters passed me, about 5 people who were given a short list of screening questions by e-mail for them to answer at their own leisure produced answers that were obviously plagiarized from various online sources.
I got suspicious when one guy presented me with two pages of nicely formatted answer complete with section headings that didn't make any sense to a question that should have required a one line answer. He'd copied and pasted the whole thing from an on-line copy of an Oracle manual, so I started doing searches of random sentences from their answers.
Another one managed to cut and paste an answer that was flat out wrong from a forum post answering pretty much the same question, and hadn't bother to read further in the comment thread, where several people corrected the answer he'd used...
It's when they can't even answer simple stuff WITH using Google it gets really depressing, and it happened regularly enough that using screening questions by e-mail was a real time saver in weeding out undesirables, despite the ease with which people could "cheat".
My favourite STL-related question to ask is simple and can be done on a whiteboard in minutes: if you didn't have std::set<class T>, but you had everything else in the STL, how might you implement a set<class T> class?
There's an ideal answer[1], and then there's people who start thinking about low-level implementations. The ones who I don't want are the ones who don't have any answer at all. I'll continue to be interested in anyone who starts talking about the way to solve the problem. What matters to me is not any code that you write (although that's interesting) but your thought process. After the candidate has worked through this, if they haven't gotten the ideal answer I always give it to them: this isn't some national secret, and seeing them process the ideal answer and realize that yes, it really would work is a pleasure.
Sadly, there are cases where it doesn't work. I had one candidate who did very well on the phone screen, but didn't know the STL. I looked up what he claimed to know (Qt) and reframed the question in terms of Qt's classes—and he still couldn't come up with any answer whatsoever.
-a
[1] The ideal answer is precisely such because you're writing much less code and using stuff that people smarter than you have written. It's to use a std::map<class T, bool>.
Not everyone has performance anxiety, but I've met plenty of programmers who do. Unless you spend the first 15 minutes of your interview calming them down and warming them up with simple technical questions that put them in the mode of programming, you might as well be asking them to sing the star spangled banner naked in front of an audience.
If they don't have a strong open source background (not everyone does), earlier in the process it helps to ask a couple very simple programming tasks, such as writing a function to reverse a string in place or the infamous fizzbuzz. It will weed out people with great resumes but poor skill.
The best interview is a pair programming session
So once again, we're back to: there is no single best-way to fully assess a candidate.
Even for juniors and new grads, at school people code next to each other all the time.
Exercise: give them a class with empty functions, and some unit tests. Ask them to write the class and add some more tests. Also, one of the tests you provide must be subtly buggy.
I had an interview once that had me complete 3 small tasks on my own: a debugging task, a database task, and a small web site. All three had to be completed in under 2 hours. I wasn't given much direction, other than the specs of the 3 tasks.
While I think this was a moderate evaluation of my skills, it ended up being more a test of if I could complete the tasks on time, vs. the quality of my work: I ended up not making it to the next step of the interview because I was too concerned about quality code over actually completing the tasks (with mediocre code), and didn't complete the tasks, thinking that they would rather get a feel for my quality vs. if I can complete a task. I did it remotely, so they never really got a feel for my character or personality.
Had it been in a paired session, things likely would have gone differently. However, I did learn more about where their priorities were, and ultimately feel I probably would not have been a good fit.
This a thousand times over. Go through their github (or whatever) profile, look at their commits, see how they write tests, etc. It doesn't take a huge amount of time, if you are any good at what you do, you'll quickly be able to filter out a failure (like, he doesn't have code that you can browse).
Not through any fault of that though, I realize that having a github account with projects and such that you work on is a great help, and I accept that myself not having one yet is a great disadvantage. What do you recommend to get started out on github? Is helping out open source programs still viable? Or now it seems either writing self-run projects, or forking others projects and helping them with those. Is there any clear starting point for githubbing? (and how do you squeeze it in after a 10 hour work day!)
Just questions from someone worried about my empty github profile :)
Actually that wouldn't be the worst idea in the world.
But every time I read "FizzBuzz", I drop everything and go back to refactor my latest version, aspiring to get it down to one conditional.
>> (1..100).each { |n| puts `curl -s fizzy.heroku.com/#{n}` }I revisited my FizzBuzz just now and threw together this obscene Python one-liner.
(If I write this kind of code in production, I should be a guaranteed "no-hire"!)
from itertools import chain, combinations
fizzbuzz = lambda n,terms: ((lambda d: d.update((x%d,w) for d,w in sorted(reduce(lambda (d0,w0),(d1,w1):(d0*d1,w0+w1),sorted(x)) for x in chain(*(combinations(terms.iteritems(),s) for s in xrange(1,len(terms)+1))))) or d)({0:str(x)})[0] for x in xrange(1,n))
print ', '.join(fizzbuzz(100,{3:'fizz',5:'buzz'}))After all this FizzBuzzing I guess I should start writing one up just so I can say I've done it.
I agree with the latter. This is how I used to hire programmers, one month on a freelance contract first with a single project. The importance of the code itself is also overrated IMHO. It's just one tiny part of someone you add to your team, what about creativity, taking initiative, never giving up, being a team player, etc. Each of them are equally important to plain coding skills.
I've got a family. I need stability... and health insurance. Unless it's my only offer, a substantially better offer, or a dream job, I'm inclined to go elsewhere.
If I'm job hunting, I've got other offers on the table. Ones that do guarantee a salary for more than a month. Ones that include benefits. Ones that expire long before your trial hire would be over.
If you try using this method, I don't know of a top notch programmer that you would be able to attract, simply because the hiring process is too painful.
As a prospective employee wouldn't you like to kick the tires before making the leap into a new job that you might find yourself wanting out of six weeks in? There's so many things you just can't know about a company until you're on the inside.
Edit: Remember that 37s has a lot to offer: Remote work with great people, top-tier name recognition in the tech world, and a long history of building and releasing open source tools. That's a really compelling package compared to some of the bigger names in tech.
You can't really expect someone to quit their existing job just for a chance to work with you.
If you have to weed out thousands of candidates, trying brainteasers to estimate their problem-solving capabilities doesn't seem to be a bad strategy. If a candidate is not familiar with a particular problem, his/her solution may be suboptimal, but the process of getting to that (suboptimal) solution may be a good indicator of the candidate's problem solving skills.
Obviously, syntax-checking a candidate's on-paper Javascript code isn't going to help anyone.
I was asked to come up with a sorting algorithm that would run on 4 computers with more data than could fit on any one computer. I was then asked to do a complexity analysis on my algorithm.
The only thing I think that really tested was how long it had been since I took an algorithms course in college.
My confusion was the job was looking for a generalist who had experience with lots of different technologies. Specifically, working with node.js and socket.io polling methods, API design, working with some python framework and backbone.js.
I'm honestly not sure what the lesson there was. Am I a bad developer? Do I navigate interviews poorly? Does my breath stink?
If an employer says they will get back to you and 2 weeks go by, it's worth contacting them and asking if they have made a decision yet. If they say yes and you don't have any clue why they wouldn't have hired you, I don't think it hurts to ask them why not. You might get a BS answer or you might get some good insight.
Just keep improving your skills, and keep looking.
I wouldn't have been able to tell you what an insertion sort is, by name. But by looking at that image, I can describe it pretty well:
Have an outer loop that counts from the first position in the list to the last position (call it n...). At each step in the loop, make sure that the first n items are sorted. This probably implies an inner loop that finds where, within the first n-1 items to insert the nth item. And given that with only one item, it's already sorted, we can probably start with n on the second item.
Yeah, I think I could code that, given the image.
Obviously i cannot show something i'm currently working on, and everytime I look back at some code I wrote six moths ago I find myself wondering "did I really write this piece of crap?!". And it's always been this way - at least since i've been six months into programming. I hope it is a good sign.
I think as long as you're applying for jobs you feel you're appropriately skilled for, you shouldn't get too self-conscience about your code. Like most personal things, it probably looks worse to you than it does to someone else. The fact that you are self-conscience about it, likely means you have enough awareness to prevent you from becoming overconfident in your skills.
Is this obvious? It's not like photographers don't have a portfolio of past project that they use to showcase their work. Unless you code under an NDA, of course.
Nope! Instead they gave me an hourlong verbal interview, which went great, took me to lunch with the team, which went great, then sat me down with a lead developer, who had printed out some of the winning entries from the obfuscated c contest, and asked me to read them and tell him what each program would do when compiled. That did not go so great. I didn't get the job.
That experience remains burned into my brain, and as a manager I won't give puzzles to an interviewee. I'll ask to see code and expect them to speak intelligently about it & the choices they made, but I won't ask them to write code (or, compile code in their head) on the spot.
(Postscript to my story -- that company never shipped a Mac email client. I believe they interview-filtered themselves out of the entire Mac market.)
Mainly because I do bring in lots of source code. I write tonnes of code in my free time. I contribute to open source projects on occasion and have released some too. It's all out in the open. All my blog posts, everything. I'm an open book when it comes to my ability, what I've worked on, etc.
Yet 95% of the interviews I've walked into the people sitting across the table from me had to look at my resume while they were shaking my hand to find out my name.
Then the puzzles come in the hopes that I will be filtered out quickly so the person interviewing me can get on with their day.
I also find that in such an intense situation my brain is hard-wired to find the "right" answer so that I get what I want without getting burned. It's kind of hard to have a jovial discussion about code and logic when you've got a lot riding on getting the job. It's almost Pavlovian.
There's always the probation period to see if they can actually write code and solve real world puzzles. I've never met the mythical applicant that could bullshit their way through a technical interview.
(Seriously though there's a lot of conceivable reasons, and while that's one I doubt it's a major one, a lip prevents it. I think the fact that a circle doesn't need to be "lined up" and that they are easier to move, etc are much more important)
I can't say for sure because the post doesn't have date (day of year but no year). I wouldn't argue smart-aleck technical shibboleths with someone who can't display a date unambiguously without overflow rollover bugs.
I have always tried to push my potential employers to use me on a small real life project. I know it's time consuming (and you end up working for em for free) but it works both ways. I get to know what kind of work environment they have and they get to know me in a real deal. I find it much more comfortable because the puzzles always seem to be a hit and go. The solution may click or it may not.
I know it's a start but if this catches in the industry I expect employers to even pay for your 30-40 hours interview.
It's great if the new employer pays you. But what about your current employer, the one at which you've just had to spend half your annual allotment of vacation days in order to audition for a new job that you might not get?
Auditioning is definitely the best way to, well, audition people. But it puts a constraint on your candidate pool, and depending on your company and your industry that constraint can be a problem. In 37signals' situation this doesn't matter - they're built entirely on open-source tools, they don't require relocation, they can hire freelancers from a pool of freelancers that they themselves created and fostered and routinely market to, they are so famous and impressive that even in a tight labor market people will line up to try and impress them, et cetera. But there are many companies that aren't 37signals, though the 37signals evangelism team is certainly trying to change that. ;)
He's obviously very talented at design (not necessarily only UI design but general program structure and framework design). But if you needed someone to write packet routing algorithms or design a high-availability database engine from scratch, that may not be his cup of tea.
OTOH, I bet there are developers who love brain teasers and puzzles who may very well excel at that type of work but couldn't implement a huge software system with many moving parts without having it collapse under its weight.
I haven't really used their products but in terms of technical sophistication they probably aren't much advanced beyond CRUD type web apps (I may be wrong here).
If you wanted to hire a games engine developer or someone to design algorithms for google then brainteaser problems might be more relevant.
I suggest this with tongue-in-cheek, of course, but it's about as effective as any other method.
1. 45mins late for the interview.
2. Quizzed a chain of brilliant engineers. Couldn't answer a singe technical question they asked. Lots of "Sorry my experience was more with X, not Y".
3. Flatly said "You guys are looking for someone else, I don't have the skills you're asking for".
->Hired because I was honest. Learned quick. Promoted every year.
Who was the employer?
Primarily because I do terribly at white board problems thanks to disgraphia(terrible handwritting), and secondily because while the questions might give some indicator of whether or not you can code hello world, or a binary tree depth first search algorithem they do a terrible job of testing your actual domain knowledge.
For instance, when interviewing for a SDET position i've never been asked a single question on implementing a DI framework, designing custom asserts, designing a test object/mock object, etc.
When interviewing for a web development position for high volume sites i've never been asked about cache invalidation, reduccing http requests, designing an ajax event system to reduce unnecsesary calls.
Etc.
I have been asked a bunch of IQ tests masquerading as mid-low level programming questions, and random trivia questions on language features.It took me a long time to appreciate the utility of something like FizzBuzz in terms of how Pareto-optimal it is.
I think it finally clicked when I was getting interviewed for my US citizenship by USCIS in 2008. In the preliminary stages of the application process, the officials had gotten us all psyched up about how there is going to be a US civics test and an "English test". We were given a 100-question booklet of Civics 101 and a lot of anticipatory anxiety.
So, imagine my surprise when the interview came around and with it, the storied, fabled English proficiency test. You know what it was? Listening to one simple sentence read aloud and transcribing it accurately. It wasn't even a hard sentence, it was something on the order of "The man walked his dog across the yard and over the turnpike". Really? Really?! I was flabbergasted. Come on, I could pass this in like 4 languages. One sentence? That's all?
After thinking about it, I realised there was more utility to it than I had appreciated, though. Yeah, I've been speaking English for 18 years, but there were people in the room who literally did not. They could not pass this test. And for someone who really cannot speak, write or understand English much, this test would be genuinely challenging or impossible to pass. At the same time, 90% of people who can pass it probably are sufficiently proficient in English to function in this society. I'm not arguing that it's a particularly insightful, perspicacious or revelatory test, but the point is that it accomplishes more than you think, from a statistical perspective, even though anyone who is even marginally literate would dismiss it as ridiculously easy to the point of absurdity.
So it goes with "a priori" programming tests, I think.
Trying to evaluate programmatic skills is surely a challenge, but I've found a method that seems to work very well -- simple walkthroughs. I'll do at least two walkthroughs of code, asking a candidate to explain to me what's happening using two different artifacts:
* Some code they've written that solves some problem
* Some code we've written that solved some problem
This has proven to be quite effective in rooting out those who look great on paper and talk a great game, but fall apart when put into action. On the flip side, I've yet to find someone who could explain their code in sufficient detail that then proved incapable once in the job.
This means that if someone gets your brainteaser right in an interview, there's, say, a 90% chance that they've heard a question just like it before (or one with a similar "trick"). The false positive rate is too high. Many people know the answers to brainteasers far beyond their ability.
If you were to ask a completely new brainteaser, as difficult as the canonical ones, but requiring a different trick, you would get a very low success rate.
Side note: there's too much criticism on HN about "what does this question have to do with my day-to-day job". This is a bullshit point. The point of a brainteaser is to test problem solving ability, creativity, etc. It may not do a very good job of that, but just because you won't be "reversing sentences in place" in your job doesn't mean that the question is invalid. Think about what the question is trying to test.
I had to deal with a crisis this week that boiled down to a poorly optimized SQL query or two. Of course, the manner in which they failed meant our DB started disk swapping and hurting our availability.
Contract to hire is something we do but it is rare. An initial try out period doesn't do us much good since the learning curve is long.
I don't trust take home assignments unless their solutions can't be googled. And it's a lot of work to create and evaluate problems of that nature. You want them just long enough, not a huge time sink for you or the candidate. You want them to demand a range of solutions so there are many ways to fail and succeed.
I agree we should not be doing riddles/puzzles. For me the most straightforward way to gauge a candidate's ability in an hour is to ask them to program on the spot and see how they act. Yes, it's a game and the candidate can learn to beat the system and oversell themselves. But the vast majority of people I don't want to hire do not go to these lengths.
The problem with this post is that it puts "puzzles", "API quizzes" and "math riddles" into one category. I can see how one can be offended by an API quiz such as "what does X do in Y?", especially if they have a track record in Y or could just look it up on the Web if they didn't know. But math riddles? Puzzles? I'd actually be grateful that I don't have to deal with technical catches and can purely focus on a solution.
One thing to consider is that recruiting, especially at a larger scale, is about minimizing false positives rather than maximizing the collective value. Sure, someone good at puzzles might turn out to be an average engineer but a bad one? I don't think so. Whereas, I can see how someone with a huge pile of code on GitHub can get stuck in a critical situation where if they write unit tests or how they design their APIs is meaningless.
At the day of my interview, they didn't really ask me any brain teasers.
They asked if I knew App Engine, I showed them my open source project for App Engine.
They asked if I knew django-nonrel, again I showed them another open source project of mine, that used django-nonrel.
That was it.
Since the day I was hired, I did plenty of good work. Now I'm entrusted with a global CMS project which is being used by hundreds people and viewed by millions.
If they had asked me brainteasers, I would probably have failed most of them, as I don't really practise with those kind of problems. But I like building web apps. And that's what I am doing right now.
If your company earns money by solving brainteasers, then you should ask them during interview.
I looked up Pascal's triangle in Wikipedia, so now I know what it is, and I could write the code tonight to handle the shit out of it, but I'd probably mess it up on a white board when commanded to perform like a monkey in front of guys in T-shirts.
I own and run a software company that I started myself. I built the first 5 successful products (not code, products - there's a difference [kind of a big one too]) myself, and I don't give a crap about Pascal's triangle. I'd probably fail every coding test on the planet, but when it comes to delivering finished, polished, working products, that 1000s of human beings will actually use without picking up a phone and calling customer support, I've got that covered.
Stay classy with your 'code' guys.
I ended up not accepting their offer, mainly because of the things I learned in the interview. But I still feel like that was a good way for both sides to evaluate each other.
Anyone who expects me to spend 3 or more hours or so on some take home test just to interview with them, or thinks that the only people worth hiring are those that "the first thing you want to do when you wake up in the morning is code something" aren't people I want to work for. I love programming, but at the end of the work day I'm going to go home and engage in other aspects of life. I wouldn't want a boss that had different expectations.
I understand what they're going for though - the companies that do it correctly are looking at the candidates thought process through a challenging situation.
While I think it's important to figure this out, I believe it's more important to know how well the candidate can do the actual job, that of sitting down and writing quality software. Brainteasers and writing modified sorting algorithms won't get this (unless the company is in the sort algorithm business I suppose) but having the candidate write code which could actually be used in the job would.
We accept applicants of any experience level. To apply, find an open source project on github, get a patch accepted to it. It has to be of useful scope with a unit test, object-oriented code only. Send us the diff, a brief work history, a reference and why you want to work for us.
If we like it, come work with us for a day, paid at market rates, and we can both assess whether it's a good fit.
Again: Google doesn't ask these questions!
Imagine your luck if you are at a Google interview and already know the Pascal triangle. You can just put up an act pretending to analyze various aspects before unveiling your grand solution.
If companies are merely using analytical ability as a filter, a SAT/GRE style exam will do better because the problem pool is much larger making it less vulnerable to preparation.
Let me explain.
On the SATs, they ask you a simple question, and if you get that right, they step up the difficulty and on an on until they discover you're a genius. If you get the answer wrong, they ask you another question of similar difficulty to the last question you got right and on and on.
Now with regards to interviewing, I like to start from the basics ... I'll start out with simple questions about languages they've used, so that they're comfortable. Syntax isn't majorly important to me, you can google it ... however I take note of how comfortable the interviewee is with the syntax of the languages they're using (especially if its one we use). So while it won't count against them, it can count in their favor.
From there I'll ask them more difficult questions. How would you implement this one thing? We have this one problem we're dealing with, how would you solve it? If they can't do it, I'll try another and another, giving them multiple chances to show that they can hang. All the while I'm poking and prodding to find out how they think about stuff ... and what makes them tick.
And then I'll go to the really hard computer sciencey stuff (which bores me to tears to be totally honest. sorry.) ... just to see how proficient they are with it. Obviously a person who eats algorithms and datastructures for dinner in addition to acing all the other stuff is someone I'd love to have if they work well with us on a small project we'll have them do.
I do the same thing with programs they've written, side projects they've done etc. To summarize, I'm just trying to find out where they are on a continuous scale from "total newb" to "could write os x from scratch if they needed to"
Basically, I think brain teasers CAN be useful, but I think we should use them carefully. Value the way someone thinks over just them just getting answers correct.
my 2c
1. Nobody gets an onsite interview without solving a coding problem first. These are designed to take a couple of hours for a competent programmer. For each applicant, we send the challenge most functionally relevant to the position they're interviewing for. Despite the negative comments elsewhere about "take-home" problems, I think they expose a lot of information about a person's coding style and thought process that whiteboard code does not. For instance, we recently received a solution to a simple graph-related question that was technically correct and usually fast, but contained at least five nested layers of loops and took forever on a certain class of edge cases. The candidate obviously understood the problem, but seemed oblivious to the fragility of his solution. You can't grade for style or really pound the edge cases in an onsite interview, because candidates usually have their hands full enough finding any solution.
2. At least two of the interviewers spend the majority of the time on big-picture, conceptual, and software design questions.
3. I split my interviews between puzzles (the 17-minute bridge, the 200m building with 150m of rope, etc) and algorithmic questions. However, I don't care about whether the candidate answers my riddles correctly or not; I'm only observing behavior and process: in other words, whether they remain calm despite uncertainty, and whether they ask relevant questions that reveal more information about the problem and make observations that narrow the possible space of solutions. (For example, in the bridge question, realizing that the slowest people should cross together is as good as a solution to me.) I do care about their responses to the algorithm questions (e.g., find a subrange of an array that sums to 0), but mainly to determine whether they have a reasonable idea of which data structures to use, and how effectively they detect problems with the rough solutions that go up on the board at first.
So I guess it might be ok to hire somebody based on brainteaser if you need a sharp, quick thinking person which will be a great asset during brainstorming sessions. I don't know.
This was right up my alley as I currently lead 3 teams and we regularly have to deal with architecture and design issues on a large application with tons of other teams involved; and I was therefore pretty stoked about the job -except that the algorithm wall stood between me and it. Not having ever done an algorithms course formally(although I read the books out of interest) I went down in flames; at least I think I did - because I'm yet to hear from them. Wanting closure I even contacted them and asked point blank if I was rejected, but I was assured that I WASNT rejected and that they'd "get back to me".
The process itself was largely the standard "how would you write <algo x> in java", which I did not do very well. One interviewer asked me to implement a Queue using two stacks and maintained that "there's a better way" despite my multiple attempts. I finally asked him if this was an actual real world problem that they faced in their company to which his answer was a glib "No, I was just testing your problem solving ability"
To be fair, however, some of the other interviewers were better.
One actually glanced through my resume, picked on one of my open source projects and grilled me on what I'd do given a hypothetical implementation constraint on a particular piece of the architecture. In hind sight it looked like he was trying to get at my algorithmic problem solving ability while connecting to something I'd built; but at that moment I was so hung up on the problem that that my software solved than how the software itself was built that I ended up frustrating him with my pre-thought solutions. He kept asking for "from scratch" solutions to which most of my responses were "but it already exists in X, so I'll just use it".
The best interviewer was a senior guy who was totally OK with my lack of algo knowledge; and very quickly switched to designing a solution to a web scale problem. He asked me for one solution, picked out the flaws in the solution using Big-O notation and asked me to refine it. This went on until we reached the asymptote that they had actually implemented in production. Along the way, he gave me more of the solution than I did to him, but I ended up with a better appreciation of the whys and wherefores of the problem; and more importantly, problems of the kind.
My general take away was that companies that use algorithm tests make the following assumption:
algorithm solving ability => general problem solving ability
However, the problem with using this definition to measure candidates are:
1. Your company problems may be solvable using existing solutions. Then the real litmus test for your candidates is how well he can assess and assemble existing components/frameworks/libraries to realize the solution. This is where attitude matters more than anything else; as does prior work and interest in such work. "What was the best project in your career and why" might be the best question to ask such a candidate.
2. Your interviewers may never connected the LHS of this equation to the RHS. This is where, IMO, questions like "queue using 2 stacks" or "manhole covers being round" come. The interview becomes cargo-cult worship at the Algorithm/Puzzle altar.
So my suggestion to the companies that still have algorithm-based interview processes is: when presenting such a problem in an interview at least tie it to a real-world scenario that you had to deal with in your work place so its not a test of how much practice I've had with Cormen and Skiena (yup read the yegge post).
Speaking of which: Am I the only one who finds all algorithm books as being too prescriptive? They all seem to dwell on the laundry list of data structures followed by the laundry list of algorithms; leaving the reader to figure out the "what to use for what" and the "why" all by himself? I'd pay for an algorithms book in the style of Polya's "How to solve it". Skiena's war stories are an attempt in that direction, but a full frontal attack on A&DS would be great.
Aftermath: I left that interview with two takeaways:
1) I had to shore up my knowledge of algorithms and data structures simply because it was the "basic tools of the trade" and there ARE still problems around that need you to dig that deep in the industry (thanks to the final interviewer)
2) Large organizations have "how to assemble software using existing pieces" issues in their software development, not "build from scratch" issues; but developer(turned interviewer) hubris will always skew towards testing for the ability to build (proxied by algorithm skills).
Either way - 37 signals notwithstanding - it makes sense to catch up on your algo skills :)
Oh, and please, please, please lets do away with the Hollywood principle when it comes to developer job rejections. If I'm rejected, tell me. I can take it.
1.) 37 signals view is that there is no point in asking programming puzzles or brain teasers; this goes for white board questions as well. Their view is that, it's just better to go over real code the person has written, and if they seem like a good fit, then why not try them out.
2.) Cletus's view is that you should ask a "simple white-boarding problem", that would weed out in theory weed out the "bad programmers", with a certain error margin.
3.) Dmbaggett's views is that solid programming puzzles (one's that are usually are take home, or one's that you do at home) are good for attracting programmers that will fit (the role of a computer scientist at ITA). However cheap whiteboard problem's aren't necessarily the best for finding great programmers.
4.) edw519 commented a while back, that written code sample done at the time of interview is best, as it provides rooms for some discussion material that will gauge if the interviewee is a great fit for the role.
[1] http://news.ycombinator.com/item?id=834513
4.)Another article on hacker news, "I Won't Hire You", got pretty much down voted for it's view, that you have to be greater than great to work at Golem Technologies.
[2] https://news.ycombinator.com/item?id=3428567
5.)There was one other article on hacker news; the article mentioned how in the end programming interviews puzzle, didn't matter and the programmers that they hired and did the best, ended up not being the most productive programmers.
6.)An article about hacking the current interview system. Apparently the top commentator timr, really thought that you should grill the interviewee to find out if they are hacking the system.
http://news.ycombinator.com/item?id=3370341
The list can go on and on.
So in the end what is the best algorithm for interviewing a programmer? Are you hiring out of fear, that you need X to do a Y job, but you don't want X to destroy your company? Are you hiring to find a rock star programmer, who can take your company to the next level? Are you hiring a programmer to do a specific well defined job? Are you hiring to fill your ego, that you have the power over someone?
It seems very hostile to get a programming job. It seems that some interviewers want you to come in and contribute from day one, which I have never seen quite happen. Why are some interviewers so harsh and dicks with the interviewees?
If a programmer fails your interview, does it mean they are truly stupid?
Is your interview really a true gauge of IQ? Does your ego play into the interviewing process as one HN commenter wrote
"The biggest problem with anything like this is the idea that 'here is some test of inherent intelligence - I am far better than you so you are inherently unable to do this thing' which is just the biggest barrier to actually trying to do something - if you think you inherently suck or at least are simply mediocre, your motivation to do that thing is severely reduced."
The general feeling I get is that as interviewers, we are trying to put down and ridicule, interviewees for whatever reason did not pass our test. Maybe in accordance with our company's guidelines or our own ego's.
Instead of seeing the glass half full with interviewee's potential we are seeing it half empty, as their ability will never be good enough.
I think with determination and practice anyone can be "great" programmer. It's sad to see many interviewer's don't realize this, cause "they don't have time."
Why I do that? The obvious being the potential hire is required to work within a large code base (often code not written by yourself) and having the ability to read and understand quickly what a piece of code does is a great skill that not many great coders have.
Today I Learned ....