Of course this will change in the future, with more interactive models, but people who use ChatGPT on the interviews make a disservice to themselves and to the interviewer.
Maybe in the future everybody is going to use LLMs to externalize their thinking. But then why do I interview you? Why would I recommend you as a candidate for a position?
Clearly, the person put 0 effort towards cheating (as most cheaters would, to be fair). But slightly adjusting the prompt, or just paraphrasing what ChatGPT is saying, would make the issue much harder to spot.
I’m not sure why anyone would want a job they clearly aren’t qualified for.
Had an interview take home assignment done by GPT and it was easy to spot after seeing dozens of solutions. Downside for the guy was - it didn’t work.
It's also why it's kinda annoying to do live interviewing trivia questions. Can I immediately answer what a partial template specialization is? Probably not, I never used them. Can I google it in 2 minutes and summarize it as as way for (often c++) template classes to bound some of the template arguments to values or pointers? Well, I just did. Should that cost me the interview? That's pretty much what I do on the job.
Jokes aside, something about LLM responses is very uncanny valley and obvious.
In the end, it's just a new way to "Google" the answer. After all, there isn't much difference between reading off an LLM response and just reading the Wikipedia page after a quick Google search, except for less advertisements.
I will say there are still some programming questions you can give that will stump the hell out of ChatGPT. In particular I took one online coding assessment where I used it and there was a question about plotting on a graph with code and calculating areas based on the points plotted that ChatGPT failed miserably at, but someone pretty good with math and geometry would find pretty tractable.
Instead, there would be tasks that can be completed using any tools available - Google, LLM, whatever. And candidates are rated on how well the task is done, and maybe asked a few questions to make sure they made decisions knowingly and not just copied the first answer off the internet.
This already exists and is called "take home programming assignment"
We’ve all got promotions by changing jobs in the last 6 months using this method.
You can be subtle about it if it’s already an area you kind of know.
People who are unwilling to say, “I don’t know, let me look into that,” are not fun to work with. After a while it’s hard to know what is fact vs fiction, so everything is assumed to be a fabrication.
1) Select All (most likely followed by the copy) 2) Type the answer 3) Make an obvious mistake when they type else block, before the if
i've actually been called out for it in a systems design interview, under the presumption i was copying my notes into another window, but was glad they called me out so that i could explain myself
It will become a skill. In 1900 you'd interview a computer (a person who does math) by asking them to do math on paper. Now you'd let them write some code or use software to do it. If the applicant didn't know how to use a (digital) computer, you'd negatively rate them.
I don't love it, but we may reach the point where your skill at coaxing an LLM to do the right thing becomes a desirable skill and you'd negatively rank LLM-illiterate applicants.
Looking at LLM quality, we're not at that point for most fields.
A candidate can do very well on personal and web project experience questions, and suddenly blank when you ask them how an http request is structured. Or what's CORS.
Then you dig further and discover a lot more thing about them that wouldn't have surfaced otherwise because hou assumed they knew all of that.
My best advice would be to never skip "dumb" and easy technical questions. You can do it very quick, and warn ahead that it's dumb questions but you ask them to everyone.
I always start interviews by asking them to explain their own projects. However, sometimes I'll find someone who's great at explaining projects they supposedly worked on in great detail, but then when given a simple coding problem they can't even write a for loop in their own top language.
I'm not sure what specific questions you have in mind, but ChatGPT is almost certainly trained on a vast array of resumes and a diverse range of profiles, possibly even all of LinkedIn itself as well as other job boards. There is little to no reason why it wouldn't be able to make up an entire persona who is capable of passing most job interviews.
Sure, the point that superior tool use is a valid job skill makes some sense, but conceding your agency and higher reasoning to a machine which possesses none of these is to my mind not going to be beneficial to a business in the long run.
Heck, at that point you aren't even measuring whether the candidate understood the question, nor their ability to communicate about it with prospective coworkers.
If there are any questions where "repeat whatever ChatGPT says" seems like a fair and reasonable answer, that probably means it's a bad question that should be removed instead. Just like how "I'd just check the API docs" indicates you shouldn't be asking trivia about the order of parameters in a standard library method or whatever.
I suspect the way to deal with ChatGPT is to allow it. Expect the interviewee to use ChatGPT as a tool. Try out the interview questions beforehand with ChatGPT. Ask questions that ChatGPT won't be good and answering, like how a calculator is useless on a physics exam.
In an open-book test, you have to know what you're looking for and roughly where to find it in the book. That implies some knowledge. With ChatGPT you could type the question verbatim and get a potentially right answer, without even understanding the answer at all. It is therefore unacceptable for use on any exam.
NOT to browse through looking for a solution from step 0.
Presumably you have tasks that you want performed in exchange for money? (Or want to improve your position in the company hierarchy by having more people under you or whatever).
I think that what will change is that doing interviews remotely will become rarer, in favor of in-person interviews.
Interviewing as a process sucks enough as it is. It should just be a culture fit filter that takes you all of 15 minutes to say yes or no to.
Technical interviews are lame and filter for people that are good at technical interviews, not people that are good at the job.
That's a bit better than proxy interviews and people lip syncing, but not by much.
I think most people have been thinking that the interviews are mostly BS with little relationship to the job, which you simply have to get through.
Many, many people will cheat to the extent that they think they can get away with it.
It's a bit like many people cheat in school. (On classes they consider irrelevant, they might justify it that way. On classes relevant, they might justify it, that passing or their GPA is more relevant to their goals, than learning that material at that time.)
I think people generally don't believe a "you're doing a disservice to yourself" argument. They choose the tradeoff or the gamble.
Personally, I don't tolerate cheating, and I have a low tolerance for interview BS. Neither is the dominant strategy for the current field.
Allow them to use the tools, with a screenshare, and adjust the types of tasks you are giving them so that they won't be able to just feed the question to the LLM to give them the completed answer.
Interviews should be consistent with what day to day work actually looks like, which today means constantly using LLMs in some form or another.
Consider that this may not be typical.
Source: https://survey.stackoverflow.co/2023/#ai-sentiment-and-usage
To be fair, the number of "Yes" was "just" 43% but that's still a very large amount of developers, not including those who plan to use it.
And I can still use ChatGPT and similar tools for some of what I do. It is a huge force multiplier.
I think it depends on whether the interviewer has agreed to make the interview "open book". Looking up stuff on Stack Overflow during the interview can be OK or can be cheating, depending on the constraints.
In this experiment, the interviews were not "open book". That said, I am personally in favor of open book interviews.
The point of (software engineering) interviews is to demonstrate how you solve problems. "Type the question into ChatGPT and do what it says" is not the "how" that companies are looking for.
The interesting part is what to do when ChatGPT does it wrong, and if the problem is not trivial and an exact solution is not available online, it is usually wrong. Sometimes, not by much, but it takes skill to notice the problem and fix it, either manually or by asking ChatGPT for it.
Same idea as for libraries. One could argue that those using the "sort" function don't know how to write a sorting algorithm, but in real life, unless there are particularly good reasons, rewriting a "sort" function would be crazy, and probably not what you want from your employees.
If you want your candidate not to use the tools at his disposal, you may frame it as a requirement. "on this test, imagine you are working on a sensitive project, you are not allowed to upload any details to a third party service, you can search the internet for generic information, but do as if your development machine has no internet access, so no copy-pasting". Or, for libraries "on this test, imagine we absolutely want to limit dependencies on third party libraries, even if it means sometimes reinventing the wheel, so only <list of libraries> are allowed".
They are probably not looking for that, because LLMs perform poorly with some kind of problems, and you don't want people who rely on them heavily.
This gives you a plan for designing the interview questions.
Then interviewers should stop setting tasks that require either a) copy and paste answer from leetcode or b) copy and paste answer from chatgpt.
Unfortunately that requires skill and awareness on behalf of the interviewers; who typically served in the leetcode wars and want their employees to go through the same.
I'm definitely looking at googling proficiency when interviewing.
Being good at LLMs is about as important - this means being able to tell when you're being confabulated at quickly, knowing what to ask and what not to ask, etc.
Unfortunately though, there are lots and lots of interviewers out there that do not give a shit about "how", but rather if you can provide the right answer or not.
Here are three questions, you have 60 minutes. Please provide the optimal solution to at least two of those questions. Now excuse me while I'm listening to a teams/zoom meeting in the background. Good luck, have fun.
However, my style of interview is LLM-proof anyways. I have "shop talk" style interviews, where I just chat with the developer for an hour or so about various topics. Makes it very easy to get a sense of their depth, and how interested they are in the job domain.
For what it's worth, I think LLMs are very compatible with the style of interview you describe here. For people for whom LLMs have become a part of their workflow, seeing how they interact with them is just one more way to get a sense for how they work and think about things. If it doesn't come up, that's find too! But I don't see any reason not to accept their usage in your interviews.
(But I do think it's key to ask them to screenshare if they're going to go that route, so that you can actually see their interaction to get that signal.)
I'm not hiring at the level of day to day work, I'm hiring at the level of when things go complex or bad.
To give an example, I'm not going to test a coders ability to write some CRUD application. I'm testing the ability when a junior developers comes with a crazy problem, that they can find the cause and solve it.
If I only test day to day work, you get these kind of developers that keep changing faulty code until it magically works. I don't want those in my team. I want developers that can figure out what is going wrong, understand it, and provide a solution. Do such things happen day to day? No.
And a person that can reason his way out of things instead of trying random stuff, sure as hell can do the same day to day tasks just as well or even better.
I don't, and I don't know anyone working with me thar is using any LLM (we pair). Some tried Copilot at some point and concluded it was useless for our use case.
Not sure on what context one would constantly use LLMs.
This narrative of "interviews should be like real work" needs to stop. It doesn't make any sense, this is not the goal of an interview.
Why waste each other's time in the interview when I (if I was the interviewer) can just ask for relevant projects or commits on GitHub of a major open source project and that eliminates the 90% of candidates in the pool.
I don't need to test you if you have already made significant contributions in the open. Easy assumptions can be made with the very least:
* Has knowledge of Git.
* Knows how to code in X language in a large project.
* Has done code reviews on other people's code.
* Is able to maintain a sophisticated project with external contributors.
Everything else beyond that is secondary or optional and it's a very efficient evaluation and hard to fake.
When there are too many candidates in the pipeline, Leetcoding them all is a waste of everyone's time. Overall leetcode optimizes to be gamed and is now a solved problem by ChatGPT.
Not everyone spends their free time contributing (to major nonetheless) to open source projects. There are a lot of great engineers that have enough work on their desks with their day job and there are also plenty of idiots in open source.
Asking for relevant projects or asking for GitHub profiles to gauge relevant projects yourself is what people were already doing years ago and it wasn't a great hiring strategy. Turns out judging a software engineers skills is extremely hard.
What would be different from spending that time making a few PRs to an open source project or just building something from scratch to demonstrate your skills?
A lot of engineers that I've worked with have never spent time on leetcode and struggle to answer the common interview questions that aren't easy or low medium, so personally I don't see it as that much different, there's time required either way. One is productive, one isn't.
I made the best work of my life (by a long long shot) to private companies closed source.
and elsewhere in these comments, we see:
I would expect candidates for programming jobs to demonstrate first class ChatGPT or other code copilot skills.
Not everyone spends their free time learning first class ChatGPT or code copilot skills.
It's interesting that this age old mantra about open source contributions being inappropriate for a hiring manager to expect because of what people do or do not do in their free time now does not apply to random other skill/experience that one must also acquire in their free time.
If a company has, for example, "git experience required" on the job posting you'll either need to actively demonstrate that you know how to use git or passively demonstrate by having a on-line accessible corpus of git-related work that shows your experience. You don't get a pass on that requirement just because the companies you've been working at don't use git and, well, you don't want to spend your free time learning git. And it is appropriate for the hiring manager to list that as a requirement despite claims that they'll be disqualifying some percentage of the candidates by including that as a requirement.
So for hiring engineers, I can understand when they hire with a certain bias. Sure, no one should be expecting us to do extra rounds, and sure, one can be a great engineer even without extra rounds, but the tendency is for that to take more years without that drive to explore in ones free time. And that's OK! I think it is fair though, if a company wants to hire a more driven/curious/exploring person.
I have 20 years experience in very high level data science work. I do not have a public git repo because I've worked at for-profit companies and I don't do additional free work in my spare time.
I'm the same, my git repo is a graveyard of projects all set to hidden.
Eliminating 90% of your candidate pool sounds like a great way to slow your hiring to a crawl.
Very few people have noteworthy and/or relevant GitHub activity. You'd probably be eliminating more like 95%.
GitHub activity also has a high false positive rate in my experience. A lot of the GitHub profile superstars I've worked with were always spending their time working on open source things or something that they could put on their profile. They avoided doing anything internal to the company as much as possible because they knew they couldn't leverage it for their next job.
It never was. No real-world job performance has ever been accurately measured by solving leetcode puzzles for one simple reason: problem solving is only ever going to be about 50% of your performance, and these puzzles don't address collaboration or communication skills.
To add one more nail into this coffin: Well, then you will eliminate 99.999% of candidates. Or more likely, you will get none. Seriously, read that sentence again: "maintain a sophisticated [open source] project". How many of those exist in open source? A few thousand at most. And there are millions of developers in the world.
https://www.youtube.com/watch?v=r8RxkpUvxK0?t=8m20s
from "Moishe Lettvin - What I Learned Doing 250 Interviews at Google"
Get your hiring done now while you can, when the economy rebounds you won’t be able to hire anyone. Also give your team a raise, because they’ll probably be the first to go once new options open up.
Or split this into “knows how to play nice on a large project” and “can readily learn new languages” if your company is willing to invest in training.
I worry though that it'll just be the end of online leetcode interviews and employers will bring people back into the office to interview.
Anecdotally: During COVID when remote jobs started going mainstream, the number of weird and scammy applications we received spiked. We also had a couple problems with people joining remote but keeping their old jobs or getting a second job (discovered following investigation of their underperformance and inconsistent availability).
When we started telling interview candidates that we'd fly them in and put them in a hotel for the last stage of the interview, most of the questionable candidates started dropping out of the interview pipeline by themselves.
In many cases we didn't actually bother flying them out. Just the thought of having to come into the office for a single day was enough of a filter.
Was it perfect? No, of course not. We had one candidate who couldn't be away from his family for medical reasons and we happily accommodated. We also had one hire who went through the whole process and proceeded to do almost no work at all for 6 months unless a manager was breathing down his neck at every step.
But it cut down on the number of problem candidates massively.
This unfortunately happened pretty often pre-pandemic/in-office. The majority of the teams I've worked on across a few companies have had one or more donothings. The bigger the team, the more unfocused management, the more likely to hang on.
Don't feel bad about firing them: you were only ever their J2.
The reverse, yes I would mind.
I had an interview like this recently that was quite pleasant, where the interviewers and I ended up collaboratively solving the problem together because we all had different approaches - I think it had the unintended effect of demonstrating teamwork and helped the interview go quite positively.
I’m sorry but lol no. There are many places where you can’t do that in an interview because there are many employers that want to test for technical knowledge, not knowledge of tools, which anyone can learn on the job and which change as fast as fads come and go.
I find this entire thread astounding in how so many people come in the defense of ignorance and outright incompetence. If ChatGPT responses were enough to do a job, then why would a company hire a human? There would be very little value add from a warm body than just paying for the ChatGPT API and automating integration.
And corollary to that: if your job is possible to do with a huge number of AI tools, then your work is likely the menial kind and you should seriously dread being laid off and being obsoleted in the near future.
If I saw a candidate using Google or GitHub search to try to look up the entire solution then I'd stop them. That's missing the point.
No chance in hell leetcoding is going away. It will be even more important, with even greater ceremony.
> employers will bring people back into the office to interview.
Nothing stops people from getting the questions they'll be asked ahead of time from insiders, like their friends or a recruiter, which is really how people have been cheating. This is how it is possible to be Google and have identical standards for years but nonetheless observe overall quality of hires go down.
In a way it's probably equivalent to students who just bang everything into their head before the semester exam and forget it all again two weeks after. (Not that that's a bad thing, it just doesn't really say anything about a candidate)
I'd take that bet. Leetcoding is something that GPT 4 is very good at doing -- and it does it faster than any engineer can type, let alone think.
Hopefully it does. If an LLM can do that, why should I have to do that, both in an interview or outside of one. LLM-assisted programming is where it's at, and there's no going back. Being able to do a leetcode isn't a good test of a candidate in the first place.
It is the age of take-home assignments that take days to complete.
Only absolutely incompetent people would want this over leetcode.
Long story short: asking them to make small changes and then tell us what would happen was a shurefire way to detect the true cheaters and not the lazy people.
I also fondly remember triggering float errors in loops so you'd get an extra cycle due to it ending in .999etc instead of 0.
Honestly, the realistic style of work that's close to how one would actually approach problems in their day to day is pretty much ideal. In my case that would be using a nice IDE, some AI as a glorified autocomplete, IntelliSense and all that as well, in addition to Googling stuff along the way, if needed.
That should be enough to let them know both how I think, as well as show how I can solve problems and reason about those solutions. Heck, maybe even give me a simple task to build a CRUD and then talk about the choices I've made, if they're serious about hiring me and want to actually see what's inside of my brain.
But of course, in many places can't have that happen - they want to put the candidates in a situation where they just have a barebones text editor and expect them to produce good results. Blergh.
that was interesting, upvoting that employer for honesty and pragmatism
Radical honesty has been a core cultural component to many a strong team, I'm glad to see somebody else mention this. There seems to be something unique about the relationship between codering and the concepts of transparency, honesty, and truth more broadly.
Or maybe that's just a consequence of version control :)
Chernobyl being one prominent example.
At least in a field like engineering where actual successful results/working output matters, anyway.
There are other fields where the same dynamics are not in play.
One cannot solve (or even avoid) a problem that one refuses to acknowledge exists, after all.
And what is worse than lies, is self delusion, even if honest. To nit pick on radical honesty, my observation is that most people won't tolerate it, plain honesty appears to be the sweet stop inmost cases.
There is no bigger warning sign than outright lying. A normal mature person would ask just before if AI is allowed.
While, wait, thinking...
So all this "cheating" is maybe a (bit delayed) response to the above trend..
The only classes I can recall that required higher-order skills were some woodworking, CAD, and CNC/machining classes I took while at an off-site vocational school alongside my required high school classes.
Example: https://chat.openai.com/share/9179ee63-6461-479b-8a76-1a7af2... (the key part of the correct answer, the word "wait", does not appear in any of the responses, and the answer to the questions about the data loss should have been a short "no" with a justification instead of the long AI rambling).
Oh, I remember questions that were ostensibly designed to detect whether an interviewee can think. "Why are manhole covers round?" "How to measure exactly 45 minutes by burning a rope that takes an hour to burn?", and others of that ilk. I believe I was presented with a question of that sort once during an interview very early in my career. I said that I don't do puzzles, and bade farewell to the interviewers.
Does it mean I'm an idiot if I have no idea why they would be round instead of rectangular?
- Using an IDE is not cheating.
- Using StackOverflow is not cheating.
- Reading the documentation is not cheating.
I would expect candidates for programming jobs to demonstrate first class ChatGPT or other code copilot skills.
I would also expect them to be skilled in using their choice of IDE.
I would expect them to know how to use Google and StackOverflow for problem solving.
I would expect programmers applying for jobs to use every tool at their disposal to get the job done.
If you come to an interview without any AI coding skills you would certainly be marked down.
And if I gave you some sort of skills test, then I would expect you to use all of your strongest tools to get the best result you can.
When someone is interviewed for a job, the idea is to work out how they would go doing the job, and doing the job of programming means using AI copilots, IDEs, StackOverflow, Google, github, documentation, with the goal being to write code that builds stuff.
Its ridiculous to demonise certain tools for what reason - prejudice? Fear? Lack of understanding?
There's this idea that when you assess programmers in a job interview they should be assessed whilst stripped of their knowledge tools - absolute bunk. If your recruiting process trips candidates of knowledge tools then you're holding it wrong.
Your ability to use ChatGPT effectivelly is highly dependent on your technical competence.
The interview is meant to measure your acquired competence, because this is the harder part. Learning to leverage that competence using ChatGPT is very easy.
I'd rather have a developer on my team that demonstrates high technical competence than one that is GPT-skilled, but doesn't know what questions to ask GPT nor how to judge its responses.
Agree.
But two challenges: if the interviewer does not make it clear that ChatGPT/SO may be used, the typical assumption is that such use is not permitted and would be cheating.
Moreover, coding challenges are typically designed for humans. We may need to design new kinds of interview questions and methods for humans augmented by AI.
I think this makes a lot of sense, but regardless if the interviewer has specified you shouldn't be using tools to help you then it is deceptive and unfair if you do.
id argue the way its being used, is. The audio is automatically picked up from the conversation, and starts generating a response with 0 user input. Ive seen users simply read off what their screen says in those cases, which is most definitely not what an interview expects from you. Using chatgpt as a tool on top of your existing skills is fine, it requires input and intelligent direction from the interviewee, this is not that.
> - Using an IDE is not cheating.
> - Using StackOverflow is not cheating.
> - Reading the documentation is not cheating.
That's not how any form of testing works.
The person taking the test doesn't get to determine the parameters of the test. Imagine a college student pulling out their cellular phone and looking up Wikipedia during their final because "Wikipedia is not cheating"
The test is also supposed to be administered to everyone on equal footing. If some candidates are substituting their own definition of cheating then they're putting everyone else at a disadvantage.
It doesn't matter what you expect or how you would interview someone. When you participate in someone else's interview, you play by their rules. You don't substitute your own.
And I, in turn, would be delighted not to work for you.
if your questions can be answered by chatgpt (or google), you are asking the wrong questions
This whole framing of "cheating" is incredibly misguided.
It's also true that interviewers have to adapt to this brave new world, and I'm sympathetic that that's difficult and takes time.
In my view, the way to do this is to ask if they're comfortable screensharing or presenting or letting me watch as they use their normal tools (which is likely to include copilot or chatgpt or some other LLM). If so, there is a lot of signal in how they use those tools, and it gives much better insight into how they work day to day. If they aren't comfortable with that, then I think it is perfectly fair to ask them not to use any tools that we can't see.
Some interviewers wouldn't mind, or would even encourage, using all available tools to solve problems.
I would say that the only thing that should be assumed implicitly is that it's forbidden to use the help of other people. Anything else should be explicitly laid out.
Having said that if some rule is not clear or evident then the interviewee should ask. And they should never be dishonest.
All these resources should be available and the candidate should get a mark for efficiency of their usage. Not using google and/or chatgpt efficiently lowers the grade.
If you know a candidate is using tools, you can then recalibrate your questions.
The latter might look like you could fake it with ChatGPT, but it'd be hard. For example, some time ago I was interviewing an admin with a bit of a monitoring focus and.. it's hard to replicate the amount of trust I gained to the guy when he was like "Oh yeah, munin was ugly AF but it worked.. well. Now we have better tech".
I guess that's consistent with the article?
One time in an interview they asked how I felt about systemd. At first I thought it was a technical question, but quickly realized he was just probing to see if we'd get along.
I got a job offer that night.
It’s probably a little hard on those that don’t know. I’m not sure if they believe me when I say they aren’t expected to know.
Unfortunately, the question is quickly becoming dated. If you’re young and started with macOS and docker containers, you may never encounter it in your career. :)
I think overall this is a great question to sus out if someone is qualified for a role.
It works fine for stuff like "give me a tutorial on how to initialize $THING and talk to it" or "how do i set $SPECIFIC_PARAMETER for $THING up".
Where it seems to fail is when you ask "how do i set $X" and the answer is "you can't set $X from code". I got some pretty hallucinations there. At least from the free ChatGPT.
So maybe add a trick question where the answer is "it can't be done"? If you get hallucinations back, it should be clear what is up.
Edit: not that I'm a fan of leetcode interviews. But then to get a government job in medieval China you had to be able to write essays based on Confucius. Seems similar to me.
I think of ChatGPT like a pretty smart co-worker. Just because they are smart doesn't mean they are always right.
Not to mention the fact that some interviewers feel obliged to ask useless cliche questions like "why do you think you are a good fit for this position" yada yada.
Not going to be surprised if picking people based on random chance (if they meet basic requirements) is going to actually be better statistically than bombarding them with questions trying to determine if they are good enough. Really feels like we are trying to find a pattern in randomness at that point.
Bottom line is that if ChatGPT is actually a problem for the interview process, then the process is just broken.
I realize there are time/resource problems on the interviewing side, but I'd be happy to have conversations that are as long and technical as it takes for an interviewer to feel like they've found bedrock.
Whether they pass me to the next phase or not, it's frustrating to spend 30 minutes or 3 hours trying to start a fire by rubbing wet twigs together and never get to walk away feeling like I've communicated more than a few percent of what I bring to the table.
I think the next evolution of technical interviews will be hands-off, talking through problems where the criteria changes on the fly, to prevent typing while talking.
Is that a reasonable assumption? I've found ChatGPT does surprisingly well on many novel DS&A questions I pose.
It feels like this is a new form of CAPTCHA. We're trying to come up with interview questions that are not too hard for a human (who actually knows how to code) but ones that expose weaknesses in LLMs.
Custom questions, by definition, aren't available online, and with no direct tutorials to pull from, the LLM has to make more inferences about the problem and will find the question more challenging.
As for asking ChatGPT novel DS&A questions, I think it is harder than maybe you'd think it is. Any question you'd think to ask likely has a tutorial for it online somewhere, so unless you happen to make up questions like this professionally (I'm paid to do this), or you have a large unique question bank that doesn't exist online (few people have this) then my instinct would be that the questions you're giving it aren't as unique as you think they are.
As a practical example, just give it a log file and tell it to pull out specific pieces of information from it. ChatGPT struggles to write code that can dynamically check obvious boundaries for humans. Recently, I had a list of times in a CSV that I asked it to pull for me ("1pm", "3pm", "9am"), and it wrote code to just grab 3 specific indices from the string. It didn't consider the need to check for 4 indices ("10pm"). It didn't think to start the check based on where commas were in the CSV, and it didn't consider looking for "am" or "pm". It just sliced a specific set of indices in the string. That's mostly because it's used to getting questions working for specific examples, but fails when you ask it to incorporate simple broader tasks into the coding interview question.
GPT is a tool which can legitimately be used to do your job.
There are so many things that GPT can't do: take decisions, find the best approach to talk with a human being, resolve conflicts between two members of the team, and last but not least explain why of a certain solution.
Is it a coding test? Pair with the candidate. See how they think. Ask yourself: would I enjoy working with this person?
And make your own decision.
The key is to be able to see the conversation, just like how you would previously want to see what people were searching or looking up on the web if you allowed that.
I would still need to get good at leetcode, just not _as_ good.
1) A couple basic coding questions. FizzBuzz and such. Can they actually solve something basic?
2) Do a real code review with this person. Share your screen and let them review code. Observe what questions they ask and the comments they leave for the author.
3) Ask some design questions. Digging in on how they would design the classes for some new product and purposely throwing a twist in there from time to time. How do they handle this new information and adapt their design? Do they take constructive criticism well?
4) Talking to this person. Are they polite and respectful? You can help someone grow as an engineer, but good luck getting them to be a better coworker if they are rude.
Stuff like a dropdown filter list, a searchbar, customizable list layouts etc. it was nice cause it lends itself naturally to casual conversations about different possible implementations and lets you sort of riff on possible solutions, probably the most enjoyable interview I've ever done.
So it's a hiring filter that only passes cheaters who have low standards.
Removing interviewees because they don't follow directions seems like a good strategy. And I mean removing them as job candidates, not removing them from the study.
It's good to have something in the interview process used explicitly for weeding out people who don't follow directions. Something like "Email us your application, and put the word 'eggplant' somewhere in the subject line. We use it to filter out spam." And then literally delete any subjects that don't have "eggplant" in them.
Interviewer: "Create a coding challenge that requires sorting a CSV file by timestamp. Require the timestamps to be in some weird, nonstandard format and describe the format. Provide a few entries in the sample data that contain a timestamp which is ambiguous."
You can 100% tell when someone is reading off a screen and not looking at you during an interview via webcam
> Cheetah is an AI-powered macOS app designed to assist users during remote software engineering interviews by providing real-time, discreet coaching and live coding platform integration.
Companies need to start conducting interviews where tools are available.
When everyone is cheating, no-one is cheating. Trying to customize your questions is just a race to the bottom, and will always be an arms race against the LLMs.
So, instead, let the candidate use whatever tools they want - in the open, and rather probe them on their thought process.
The negative assessment of one of the interviewers about a candidate how “he hadn’t prepared to solve even the most basic LeetCode problems” is especially telling.
Maybe the candidate had really honed their sudoku solving skills instead.
ChatGPT (or local/hosted LLMs) should be tools available at workplace nowadays.
Interview while using LLMs, wikipedia, google, SO, o'reilly or whatnot should be not only allowed but encouraged.
Just have conversation/pair programming like session with gpts open and shared - just how you'd work with that person.
That's how they'll work for/with you.
Mission. Fucking. Accomplished. [0]
For senior+ candidates I honestly think the correct approach is to just lean into it though.
Encourage them to use ChatGPT at the outset, and select questions that you've already fed to the prompt. When you ask them the question, you can show them ChatGPT's output on a screenshare. The candidate can then talk you through what they like about the answer, as well as where it falls short.
A senior-level developer should almost always be capable of improving on any response given by ChatGPT, even when it gives a good solution.
And if they're not able to give better output than the current AI tooling, it's a pretty good signal that you'd be better off just using LLMs yourself instead of hiring them.
