Was there ever any post-mortem written on Starfighter? If not, would 'elptacek, 'patio11 and 'tptacek be willing to provide any details?
Slightly OT: everyone feels hiring is broken, can you list some things that are annoying from the employee and employer perspective? Here are some points:
- the process often stretches out over weeks and often months
- job posts often get 100s of applications, a lot are low effort applications, it just muddies the water for both sides
- ATS systems/job boards are annoying with the need to create an account on many sites, some forms have more than 20 questions, often asking what's already there in the resume.
A question to everyone: What would a good application process look like? For me, it should just solve the above mentioned problems. I send an email with my resume, a few sentences about why I might be a good fit for the role/what interests me about the company. The jobs@<company>.com email address could be linked to some Saas product which makes it easy for the employer to go through the applications and further communication about video calls or take home assignments or whatever are all in this email thread. The employer can set the stage of the application such as 2/5 or whatever, they can mark it as rejected or accepted after all rounds to trigger automated emails etc. Is there any Saas like this? (I can build this in a week if it doesn't exist, but no clue how to market it/get users, any pointers in case me or someone else builds this?)
Unfortunately hundreds of other people found it just as easy.
This is why I never blindly submit my resume to an ATS. I only did it this time because I was unemployed (for two weeks) and had nothing better to do.
The five opportunities that led to interviews were still based on good old fashion networking
1. A full time offer (didn’t accept) and a side contract (did accept) based on referrals from former managers
2. Two interviews based on reaching out to the internal recruiter for positions I was objectively the best person on the market for - I was a major contributor to an open source official “AWS Solution” they were implementing and modifying. I accepted an offer.
3. Responding to a recruiter he reached out to me. There was some miscommunication about the role. The team thought the client wanted someone “local” to the US since their whole team was in India. They wanted someone local to the city - I declined.
Out of the 100 applications I randomly spammed, most didn’t get around to even looking at my application and only four actually downloaded my resume.
Especially because lowering the barrier usually means getting more low effort application.
People either put a lot of effort on few applications and invest significant amount of time in each one hoping to increase the chance they get picked up; or they put a lot of effort on reaching out as much as possible, knowing they have a very small chance on each application but playing on volume.
Most recruiters are only interested in the first category of candidates. Having a long initial process is a feature. The goal is that the second category considers that given their chance of getting picked up, it's not worth filling the form.
Their live coding questions were
1. Either practical , or core CS stuff. No fancy algorithms, just basic stuff you'd get in a CS 101 DS&A class.
2. Collaborative. No one cared about perfect code or syntax, it was very relaxed and dare I say fun (if it wasn't for the nerves).
3. System design was practical and a good back and forth.
The interviews were challenging overall, but nothing they asked seemed unfair or about memorization either.
Certainly coding under pressure sucks, but I don't think there's a better way.
A complete overhaul of credentialing, so that when I have a degree in computer science from a top university and decades of industry experience, employers don't need to independently confirm that I know how to reverse a string or traverse a tree or define what XSS is.
Also, make a time machine so we can make this change 30 years in the past, if you wouldn't mind.
Similarly lots of people with significant industry experience… also can’t code.
> know how to reverse a string or traverse a tree or define what XSS is.
In my experience, it is absolutely not safe to assume a correlation between these two sets of traits.
But 'top universities' don't know how to teach CS. That's the point. Don't waste your time there.
Maybe we should have a 'bar exam' type system, but the university system is a joke at this point, and there's no point trying to put lipstick on a pig.
But 'top universities' don't know how to teach CS. That's the point. Don't waste your time there.
Maybe we should have a 'bar exam' type system, but the university system is a joke at this point, and there's no point trying to put lipstick on a pig.
I work in a top company with infamously difficult interview. I made major contributions to several opensource projects that modern tech runs on. I can easily show a clean standalone pull request attributed to me that introduces complex multithreaded code. That code runs daily on millions instances.
Why the hell do I need to spend hours answering “what a mutex is”? Also, understand that a lot of stuff is much less obvious and clear cut (e.g. how HTTP protocol works) for me than for your engineer. They only saw the basic case. For me the answer will be “it depends”.
Please, respect the candidate time and experience.
A lot of applicants will lie or bullshit something similar. How would you suggest an employer distinguish someone like you from someone pretending to be someone like you?
As a data engineer I'd like to have a chat about the pain points of the current team and contribute. I think a coding test is OK too as long as it is just one round and actually test something meaningful.
Can you reduce hiring an airplane engine designer to a set of tests? Or a custom home builder? If so, why is there not just a stack ranking of best ones, or an algorithmic way of hiring one?
Software engineering at elite levels is an art. It is conducted by master craftsmen. On top of it the best people are often quirky / weird and the social aspect when communication of needs is so essential makes it even more difficult to hire "the best".
The business models of recruiting is always either like sales (recruiting firms) or marketing (LinkedIn). I would stay away from the recruiting industry as a software startup - difficult to make any money here in a novel way.
I guess it's a hard thing to certify when the landscape itself is constantly evolving and skills are obsoleted in 3-4 years. It seems even the pace of change is itself speeding up, and every year I see more technologies come to popularity (if not quite maturity) than I can even remember, much less learn. And it's not just languages and frameworks, but entire architectures... today's best practices are tomorrow's tech debt.
There should always be room for innovators to push the bleeding edge forward, but many businesses don't really need that. Instead, if there was just some shared certification for "good enough engineering" that can deliver a reliable, maintainable product somewhat on time, reproducibly, hiring and job-hunting would be so much easier. I'm so tired of trying to charm recruiters, impress managers, placate HR, all while hoping against hope that the actual dev 2 or 3 interviews later happen to have some overlap in languages, toolsets, shared pain points, etc. It's all such a dog and pony show, with many teams having ill-fitting skillsets and cultures between members because nobody really knows what they're signing up for or getting.
And in general I've seen little correlation between credentials and quality, both in the software industry and elsewhere, so I don't think any such certifications will really solve anything either.
Software engineering is building a factory. The inputs are data. The outputs are data. All of the stuff inside the application are the different conveyor belts, elevators, gears, rollers, pounders, etchers, etc. to move and manipulate the data to make the output valuable. The factory can be constructed infinite ways without any care about physical dimensions, and the raw material can be duplicated to infinite amounts. The situation is beyond regulation and it should not be regulated. Only the product output should be regulated.
For very small shops willing to incur legal risk or limit the application pool, you can do take home tests and/or lean on side projects, with an hour of explaining the nitty gritty details and choices. At scale it becomes untenable, because you open yourself up for discrimination lawsuits, and you can't convince a lot of people who already have jobs and families to do lengthier take home stuff.
If I had to bet, I'd bet some day it ends up like plumbing. You work as an apprentice, learn real stuff on the job, and then there's some kind of state certification which itself is a joke, but requires you to have gone through an apprenticeship program already where all the real shit happens.
Companies trying to hire "the best" or whatever are still going to have problems, so as long as there's not much regulation increasing cost of dev and uniformity by an alarming degree, we'll still have insane hiring (and insane pay too, regs will make it unprofitable eventually).
Instead I'm always surprised more places don't rely on references and prior experience. Yes people lie, but in my experience its relatively easy to tell the difference with a simple glance at their LinkedIn. If I look at a candidate that spent 2-3 yrs as a Software Engineer at some company that I'm relatively familiar with, and they seem to have 2nds and thirds to people I know in the industry, then pretty good chance they aren't lying. Same for people taking the time out to write recommendations for them. Even bigger signal if they want setup some calls with their prior co-workers who can vouch for them, to me that means they stand by their work and reputation.
I recently went through about seven rounds for a senior role. During that time I repeatedly offered to setup some time with my prior coworkers from those I directly managed, to peers, to those I reported to (executive team). My thinking being that they could hear from the horses mouth how I lead a team, my work ethic, etc. They did not take me up on the offer, which to me was crazy. Yes, I could be running some machiavellian scam with 2-3 people who also made a fake LinkedIn, I also could have put up fake articles in PR Newswire announcing my last position, could have spoofed all those blogs I co-authored from the company I worked at 2 jobs ago, etc. But really, wouldn't it be a better signal for a candidate to offer and have all these things?
Instead you see an industry that puts someone with ~10 yrs experience through a whiteboard interview. It makes no sense.
At the start of a SWE career, I don't think things broken. Study up some leetcode and write code on a whiteboard to prove that you, as a candidate, are smart and motivated enough to play the game. It's after that, that things get out of sync and you get absolutely amazing people who have written world changing software who can't pass your stupid puzzle of a whiteboard interview. And then what do you do? This person failed your test but you want to hire them. Here's the part where we fail, where we are too stuck on following the rules.
They failed the test so you can't make an offer? No. Move heaven and Earth to bend the rules, and make them the offer anyway. Don't hide behind corporate handbooks and the HR or the legal department.
Obviously this power can be abused, but I don't take it as a given that it must be, given a hiring panel.
Google could have made an offer to Homebrew's Max Howell but chose not to, which speaks volumes.
At the end of the day, programming is just a writing job. So if you apply for non-technical writing jobs, e.g. to be a columnist at The New York Times or The New Yorker, do they not read any of your preexisting work on the grounds that it could open them up to discrimination lawsuits? Similarly, do folks who hire designers typically not look at their portfolios?
Fake it to you make it is actually a real thing. Though you actually have to be able to make it. You can overstate qualifications and experience but not competence. I don't lie about anything but i've definitely known people who have lied in order to get hired who have gone on to perform extremely well.
Is that because when looking to make a possibly subjective judgement on the performance in a test and especially what a side project shows, it then becomes more difficult to prove that the judgement was not instead made because of some protected characteristic of the candidate?
My read on the approach is that it works very well for certain types of engagements, but other types, such as where you are trying to build a solid team of peers, requires something _closer_ to the traditional methods. That being said the Starfighter approach is probably right most of the time -- teams are almost certainly never stable, management usually treats ICs as fungible cogs and moves them around, in which case individual value should trump things like team fit or culture. But we're human.
Appropaches like Google's work better (n.b. I am far from a proponent of what they do) because the interviewers are human and despite the whole blind-panel thing if the interviewer and interviewee can't get along the interview won't go well. So culture and team fit _is_ evaluated significantly more than something like Starfighter. People care a lot about that implicit bit, in my experience.
I tailored the work sample test to the role. I did it for the compliance hires and for the project manager. The results were unexpectedly excellent.
As a result of this process, and having a solid company culture, there are around ten or twelve folks that have a new career in the security field that would likely not have had another opportunity in the classical hiring scheme.
I think the state of the industry with respect to hiring and evaluating folks is broken across the whole spectrum. It is like once someone becomes established in a field, they invent all manner of gatekeeping tests that they didn't pass to get where they are. The whole story of sabermetrics is another example.
I did a lot of silly things after winning the initial play through.
- Solving the final challenge using a basysian spam filter on trade directions triples.
- Live 3D market visualization in Minecraft
- Buddy of mine built a stock exchange stack, speaking the same protocol, and we had PvP contests on who could market make the best on it. I won with a PHP powered marketing making bot with live code reloading.
- Built a cardboard box out of scrap electronics from adafruit that would let me adjust a market maker bots net position by tilting the box from side to side.
- Was either #2 or #3 to solve all the challenges
On the business side, if I recall correctly, most companies wanted to just shove the winners down their existing bad hiring pipeline process, and a lot of successful contestants who showed up already had great jobs, and so weren't actually looking.
I don't think very many players did all the Starfighter trading challenges. patio11 or tptacek might be able to confirm, but I vaguely recall being one of like 100 or so to do all six levels of out of 10,000 that started. But I also had free time for playful coding and was nerding out on online challenges back then.
When Starfighter came out, I was energized and hopeful that hiring would go in this direction for programmers or that it would open doors to me as a job seeker that I couldn't every push through. No such luck.
From the hiring side, I have been on two hiring committees this year. Most applicants lacked any meaningful programming projects they could use to demonstrate skills. Something like completing a series of online challenges would be a difference maker.
That said, one candidate mentioned doing HackerRank challenges to learn Python and SQL, but totally flamed out on being able to even talk through a for loop in Python or come up with a simple select against a table in the in-person interview. Maybe nerves, but it also felt like the candidate's experience was 100% as just a warm body in various cubicle farms. It was very awkward. Several non-developers on the hiring committee felt like the candidate had actively lied about previous experience . . .
This tends to be especially bad in companies that use a matrix organization where your department or career manager is in a totally separate hierarchy from the project-level managers you actually receive taskings from because the latter know your performance traits but the former decide where you get assigned to work and whether you can be fired or not, and they tend to have a huge incentive to staff critical programs, even if "staff" means fill in with net-negative warm bodies, because that's at least faster than filling with new hires, which also may not even be possible when you have very specific location and security requirements.
As far as I remember, at the beginning and for quite some time the Stockfighter trading challenges had reliability and performance issues (server capacity). So I decided to wait a little bit for the product to become more stable, and then get into the challenges. Unluckily, at that time Stockfighter was already decided to become shut down.
At least 30% bombed writing a for loop and if statement though. And we let them pick any language they wanted, so that wasn't it.
patio11 once said that the code wasn't in an open-source ready state.
There were some folks who went so far as re-implementing/duplicating the backend trade simulator in other languages. Some of those were on github . . .
* Our go-to-market was contingency recruiting; we'd get paid for placements. Our CTF and HN calling cards easily got us added to the recruiting funnels of big tech companies (I'm assuming any of you could have done that too; I'm just saying, we never had to cold call). But it didn't change anything about how those companies qualified candidates, so to make that business successful we'd have had to do the same dialing-for-dollars matchmaking work any recruiter does, with comparable results.
* We took things very personally. You build relationships with your earliest users (candidates, in our case). Recruiting funnels at our clients had no such connection. The impedance mismatch was demoralizing; our lived experience was that every time a candidate we presented to a client got rejected, we had ourselves failed (I mean, really, we had!). Similarly, the idea that we were just another hoop that candidates had to hop through (but only the non-name-brand candidates who didn't know how to skip us to get an interview at a client) grated on us.
* A lot of second-system-syndrome on the implementation side; we'd done successful recruiting CTFs before, and set what was in retrospect a completely bizarre goal of outdoing ourselves at that job. That mistake is all me.
* Japan/Chicago remote founding team was very difficult, most especially for Patrick. There are teams that could handle that, but their members are more punctual about routine meetings than I am. Both the US and the Japan side felt checked out to each other. This didn't kill us; other things did, and we could have pushed through this problem. But it made things miserable for everyone. Again: this is all me.
* Erin & I were self-funding the company, which was expensive (just in terms of paying everybody's personal bills). Towards the end, it had become clear that we weren't going to drive 3 FTE worth of revenue on contingency placements in the next 12 months. We could have switched up the business model, driven user growth metrics, and gone out for seed funding, but none of us at that point were convicted enough of the business to stake our reputations on that. For all 3 of us, our time was worth too much. I think we made the right call.
It's been almost a decade since we did this, and those years haven't been especially kind to recruiting startups. There are knobs we could have turned to keep things chugging, and probably chugging comfortably, for many more years. But ultimately, I don't think there were any significant rewards to win by building a tech-company-specific recruiting firm. I think the company was ultimately fated to wind down sooner or later. If you're in that situation, sooner is always the better option! Life is short!
The underlying idea behind Starfighter, about the right way (or, at least, a right-er way) to hire people, has proven out repeatedly since then. We hired resume-blind and with work-sample challenges at Latacora, the next company me and Erin started, and we hire resume-blind and with work-sample challenges at Fly.io. Recruiting this way is a competitive advantage. The degree of advantage varies with the market; when the market is weak you can hire good people with terrible processes. But the advantage is never marginal, because bad hires are expensive, and there's so much hidden talent out there.
Turns out, finding/discovering hidden tech talent is relatively easy.
Finding the "Economic Buyer", a true hiring executive with budget and authority to make timely hiring decisions gets worse every year.
Bureaucratic Red Flag, when HR Bozos start 'adding value' to the process.
As companies grow, they mindlessly get in their own way.
> We took things very personally. You build relationships with your earliest users (candidates, in our case). Recruiting funnels at our clients had no such connection. The impedance mismatch was demoralizing; our lived experience was that every time a candidate we presented to a client got rejected, we had ourselves failed (I mean, really, we had!). Similarly, the idea that we were just another hoop that candidates had to hop through (but only the non-name-brand candidates who didn't know how to skip us to get an interview at a client) grated on us.
That sounds rough.
If you could do it over, do you see a way you could get more buy-in from some clients, such that they commit to "advanced placement" in their process, to candidates you provide? Or are pretty much all companies going to have their entire hiring process set in concrete?
What about interfacing at the hiring manager level, at companies where the hiring manager normally has some discretion to vouch for the candidate and bypass a lot of process? Could you get sufficient trust with those? Still the rejection problem?
If I was going to do it over again --- and, again, I wouldn't, because I feel like I also learned this just isn't a great business to be in --- I would have done something much closer to "Stack Exchange, but for CTFs". Run a couple of pure investment years, build up a base of users and some public profile, and then do lightweight recruiting stuff (probably just curated job ads for awhile) on top of that.
Got a short list of companies in addition to Fly.io that are using a hiring process you like or love?
It's very hard to shift companies that are eigher larger or with more internal history than a startup to try something like this.
Each level had an increasing level of what you needed to do, with the first level being to buy some stock (which you could do from the UI).
Most levels had you operating some kind of marketing making system, where you had to constantly update passive positions that made money as bots traded against them, and the prices shifted.
The last level required you to "hack" the system to get info you weren't supposed to have, in order to identify the insider trader in the middle of 50 other bots. It was quite different from all the other levels.
Here was the level solving code from my second run through:
I read it as an homage to HN days of yore.
For anyone who didn't see the '80s light sf movie, The Last Starfighter, the premise is appealing and classic: (minor SPOILERS) young adult kid stuck living and working in a trailer park, wants more, there's a Starfighter video arcade game set up in the park, one day he goes for the record on the game, and turns out the game was a recruitment tool for something much bigger than his humble trailer park, and he passed, rises to subsequent challenges, and becomes a hero.
(Co-stars Robert Preston, as a very enterprising and scrappy recruiting headhunter, a bit like his The Music Man film character, but without the musical numbers.)
(It seemed to change, after the dotcom boom, when the jobs turned into a lot more money that attracted a lot more people, and class barriers were erected to many of the better jobs. There's still a chance for upward mobility, despite artificial class barriers, but people will tend to burn their energy mimicking the shibboleths and practicing for the hazing rituals, rather than on things that would attract and teach them. Which brings us back to this application domain, of helping companies to hire people who would be great software developers.)
https://en.wikipedia.org/wiki/The_Last_Starfighter
https://www.imdb.com/title/tt0087597/
"Greetings, Starfighter. You have been recruited by the Star League to defend the Frontier against Xur and the Ko-dan Armada."
Just imagine how good that could've been though
We started with this dream of completely revolutionizing the way people hire in tech (with hopes of expanding to other fields).
First of all, we learned that for 99% of companies they don't have a screening problem, but they have a sourcing problem. Our original company was along the lines of HackerRank / LeetCode / etc, and created due to the founder's experience as a manager at Amazon where he'd get 1000s of resumes that needed to be screened and the process was eating up too much dev resource. A product that solves this problem is appealing to companies that get 1000s of resumes per posting, but basically useless to most companies who get very few candidates, and even less appealing to recruiters who care much more about top of the funnel than the screening process.
After that we pivoted to an event format. The idea being that we could attract 100s of applicants to do a challenge in realtime. This reduces cheating risk and allowed us to act as a means of sourcing candidates (instead of just a screening platform). The idea was 100s of candidates would enter and the companies we got to sign up would compete for the top applicants. It sorta worked. The platform was good, companies were interested, and we got some people placed.
HOWEVER, the profile of people that will do a multi-hour open ended challenge for the chance to maybe get hired by some companies largely leans toward entry-level people. We had some experienced people that did it because they were bored or just curious, but generally seasoned engineers aren't going to waste their time, and forget about "passive job seekers" which is what all recruiters want most of all.
Ultimately, the company ended due to a lot of personal issues between the founding team, but the product itself was good, and the market fit was decent, but far from perfect. I'm pretty sure if we would've kept it together we would've eventually became very similar to HackerRank just offering a platform where people can conduct the same crappy tech interviews we've been using forever.
Wasn't this assumed to be the case going it to it when you discussed the business cases? It's hard to imagine any other result would be likely.
Which then puts any candidates in the position of, why should I spend time and effort on this little game if it doesn't lead to any advantage at all in actually getting good positions?
And so, the only people who messed with it were developers who had the free time to play with stuff like this and didn't particularly care that it wouldn't lead to any advantage in getting jobs.
Which means, it's a cool toy that was high-effort to build and probably to maintain, but had no viable path to earn money from anybody.
> Which then puts any candidates in the position of, why should I spend time and effort on this little game if it doesn't lead to any advantage at all in actually getting good positions?
This is the same position TripleByte was in. They advertised themselves to developers as "show us you can code", but they advertised themselves to companies as "we will find candidates who pass your existing hiring process". And when pressed on which of those things they were actually trying to do, they admitted it was the second one.
If TripleByte really succeeded with this, the customer companys could save quite some money by simplifying their hiring processes. The problem in my opinion rather is that most company's hiring processes have an official and a hidden agenda, and TripleByte only attempted to find candidates that might succeed in the official agenda.
https://www.kalzumeus.com/2016/12/30/kalzumeus-software-year... says the same thing.
Did that ever happen?
I’ll almost certainly buy it, though.
Edit: patio11 called it Stockfighter, but others didn’t? Here’s the original launch post: https://news.ycombinator.com/item?id=10724592
And it was only purchased because Lockheed paid millions in bribes to officials for which AFAIK none of them (nor the company) were ever punished.
So in terms of post-mortem I was thinking of something official like that :)
Ps its successor the F-16 was a lot better though and is still going strong today.
Wow, I thought you must be exaggerating, but it was literally hundreds. Some air forces lost nearly half the planes!
Added to that were inexperienced crews and well, disaster ensued :P They called it the lawn dart or widowmaker in Germany.
I think this is why the F-16 was so much more popular in Europe, it was actually designed as multirole and was and is good at it.
https://www.kalzumeus.com/2016/12/30/kalzumeus-software-year...
I think I might've been the one referred to as submitting the excel file with the explanation... as I think that was how I solved the last problem on the stockfighter thing.
