They conflate domain expertise with coding expertise, and then assess that people with domain expertise demonstrate great success at coding tasks, which suggests coding agents are so good at writing code that domain experts can now cut software engineering experts out of the loop entirely. Yet if you look at their classifier, it classifies user expertise almost exclusively according to standards that measure expertise in coding. No wonder it predicts success at coding tasks. This just in: people who know how to develop software are better at developing software. What a fucking joke.
I didn't read it as such - I read that people with expertise have more success in reaching the goal of the session. Still your point stands = how is this news?
What baffles me, is that expertise of writing code is not important and is hand waved away = non-technical person can reach their goal and the session will be deemed successful. They don't conflate that domain expert = developer, and they dismiss that expertise (ie. updating legal rules matters, not how they are implemented) so i'm confused. Shouldn't both matter?
I would normally assume that my understanding of the problem I'm trying to solve is a blend of domain knowledge and system knowledge; that is, some of the what and some of the how. And just as important as bringing domain knowledge to the table is bringing the ability to articulate it with conceptual clarity in a way that bridges to the software. I have met many very deep domain experts who don't have that skill.
My personal interest of late is in capturing domain knowledge and managing it like source code, and using it to share with both the humans (in visual form) and the agents (in plain text form) some of that foundational understanding to make them both better at both the what and the how. This seems aligned with the article when it says "the more understanding a worker brings to an agent, the more quality work the agent is able to do" but I would also say that the more understanding we can bring to future humans new to the domain has pretty good ROI as well.
A more useful headline (that fits in our limit) might have been the takeaway they want you to have:
The more domain expertise brought to Claude sessions, the more end in success
