m3h

When I look at what engineers and non-engineers are doing with LLMs (Claude Code and friends) at any company, I'm finding more and more instances of busywork.

Burning tokens is equated with making progress. More conversations are treated as more "issues" handled. Coding sessions have changed into speccing, PRD, test plan, code plan, code generation and review pipelines. Only for every single piece of artifact to be double-checked by a human. This is considered "agentic engineering". And agentic engineers' token maxing is becoming the norm and is treated the same as "employee performance/efficiency".

It is like we've handed every engineer (and non-employee) uncapped credit, burning at dozens of dollars per minute. No one is talking about whether the costs are justified. No one accounts for the tokens (and money) spent. All for what feels more and more like busy work.

Am I the only one seeing this?

Recently, I've found that the candidates in my company's hiring pipeline are using ChatGPT to solve the coding test we use for screening.

While the coding test isn't necessarily complex, it is still hard to finish it within the time limit I provide. This means that candidates who are genuinely attempting it often submit the code with some tests failing. However, some candidates submit the tests with all tests passing, which would be pretty tricky (but not impossible) in the allocated test time. Sometimes, there's a tell, like their submission time being too fast to be reasonable, but it is not always possible to tell.

I am using a code-screening test platform (CodeSubmit). Despite having challenging problems in its library, it is not ChatGPT-proof. But I'm also amazed at how well ChatGPT solves the problems (well, most of them). It seems to be able to follow complex instructions and adjust the program to pass unit tests.

Are you seeing the same problems in your hiring pipeline?

How can I ChatGPT-proof my screening tests and take-home assignments?