Getting an inscrutable Russian Nesting Doll response from the erstwhile retail sales manager of a company asking you to set up Nomad on top of Nomad and ELK on top ELK is an off-putting experience, but it's the kind of off-putting experience we're comfortable giving people: what we're talking about is either going to click, or we're probably currently not a good fit, and that's fine.
On the other hand: the experience you apparently had, of going through our process and being unceremoniously jettisoned without explanation is off-putting in a different and less tolerable way. It's something we try hard never to do, but we're a small team (much smaller in 2021) and we screw up sometimes. I apologize.
Edit: in my experience, the code challenge made a lot of sense.
I put the experience on GlassDoors, but in general... it's fine, but took way too long.
On plus side. The coding challenge was amazing and I learned a lot doing it. I passed the first code challenge, then I didn't pass the followup exercise which was on slack. It was very clear that the exercises are directly related what I would be doing.
On the negative side, the whole process took far too long and there were around 1 month delays in all the messaging. That made me think I don't want to apply there again, because I don't have infinite time in my life. However, if you can afford to wait, it is fine.
(I'm writing a post about the iterations we've done on this process since 2021.)
Long story short: you're right; you had an experience with us that we're not OK with.
The actual interview was really fun (although it was at the end clear you are looking for someone a little different) and I learned about eBPF, which was great.
If you fixed the delays, then it's even better.
I asked about leveling since the fly.io hiring page claimed to be hiring levels 1-3 and I didn't see how I could be screened out of consideration for level 1. They did not reply.
We do respond to responses we get after rejecting people, but it can take us a few days; we tend to do batches of them all at once. My take is: while people are still in consideration, we have an obligation (we don't always meet) to be responsive, because people are pending their career decisions on our own decisions. Once we've communicated that decision, we're less obligated to be zippy, because you have the high-order bit of what you need from us.
I don't know anything about your submission and deliberately didn't try to look it up, because our grading of your work sample is nobody's business here, and I don't want to create the impression of communicating anything about it.
Having said that, I can tell you what we tell everyone who asks why their code sample didn't pass:
We've made a decision to budget the amount of time we ask from candidates, end to end, from application to final decision, to the amount of time they would spend in a hiring loop at a typical company that just did interviews and resume screens.
That implies that we have to keep our coding challenges short and succinct. I can't ask you to spend days coding something; that's more than an interview would ask of you.
A consequence of a short coding sample is that there's less surface area for us to derive signal from. We have to be pretty greedy about selecting coding decisions to judge, because there isn't all that much code to work from (for anybody who doesn't know, our current platform code challenge is to write a TCP proxy).
A consequence of having to jealously collect signal from coding challenges is that we're limited in what we can tell you about our grading. We can't offer you the rubric we use (there is one; reviews are mostly blind, and done by a pool of engineers), because that deprives us signal. We'd learn how good you are at following directions, but not much about how you approach systems programming problems.
I'd want a scoring rubric (and fast responses to emails!) too. I see where you're coming from. All I can do is tell you where we're at.
I had been receiving very quick responses but haven't heard back in a bit over a week.
Last email said I was moving to the final round for the platform position. It said you'd follow with a couple more emails with further instructions but haven't received them. Sent a couple of pings over email.
Reaching out over HN in case this is a bug on your hiring tools or my email setup. I understand that sometimes things get busy behind the scenes. Just need an ACK to know if I'm still under consideration and make better informed career decisions.
How does that make sense? I was told that you got a ton of applications and that I didn't meet the cutoff for the pool. Surely you can't just keep the top X (open positions + margin) and try to place them into appropriate levels if they make it through the remaining rounds? You very likely won't have an appropriate mapping of skills to your open roles and candidates will be left dissatisfied, right?
Isn't a hiring challenge supposed to be a more objective assessment of engineering skill? If there's less "there's less surface area for us to derive signal from" then maybe it's not better than the traditional model. To be told that I didn't do well enough to be considered for a junior position sucks because I'm a mid level engineer.
I'm certain that my proxy worked well. I was able to push ~7 GB/s through it spread across 1000 connections on my local machine. It can do that with reasonable resource utilization, good latency, no race conditions, and no errors. I also wouldn't say my notes were skimpy. I could have written more but I wanted to be respectful of the reviewers time.
In fact I would rather you openly shame my submission here because at least then I would have the oppourtinity to reflect and improve. I hope this level of criticism isn't too much. I just wish that software engineering interviews could be more rational.
I will say though, they were very polite.
We try to be clear in our assessments (and to give some feedback about our decisions). I feel like we do a better-than-average job of being transparent, responsive, and fair, but I would feel that way: I'm the Retail Sales Manager.
We're happy to take the credits back if they make you uncomfortable. (I kid. I kid.)
I didn't clear the work sample but man, I had fun doing it. I learned a ton. I most appreciated that they gave me legitimate feedback as to why they decided not to move ahead with me. It highlighted a gap in my current knowledge/experience that I was glad to discover.
That being said, it did take a very long time to get a final answer back from them, but overall I'm a happpy reject. Could even see myself applying again some day.
Thanks for bearing with us!
Out of curiosity, what did they send you, and what do you find weird/off about the process? It's definitely not the norm [0] but isn't it better than spending a bunch of time working through back-and-forths in "normal" hiring pipelines?
