We are talking about hear say anecdotal evidence from some influential people in the industry. The people mentioned in the comment I responded to have influence to organize certain research. Some measurements (even if not ideal) can point to 20x vs 0.1x speedup differences at least.

I indicated that there is at least some research pointing that developers (experienced or not) often overestimate the gains of using AI. There are a lot of other things that may prompt people to say things regarding emergent industries, for example investments into the AI industry.

I am interested if the claims are real or perhaps overstated. Therefore I asked what kind of information this is based on. This is how science works compared to marketing claims. Hypothesis lead to experiments that result in measurements that lead to a conclusion.

But as of now I still didn't even get a link to the statements supposedly made by these influential developers, this is the rhetoric with a lot of claims around AI especially. And therefore I am still skeptical about such claims until I see some concrete evidence.

So I would say yes it is fair to ask if they measured their results to back up their claims, especially if they are influential developers.

I think in a small conversation like this, it's probably not entirely fair.

However, we're hearing similar things from much larger organisations who definitely have the resources to do studies like this, and yet there's very little decent work available.

In fact, lots of the time they are deliberately misleading people (25% of our code generated by AI being copilot/other autocomplete). Like, that 25% stat was probably true historically with JetBrains products and using any form of code generations (for protobufs et al) so it's wildly deceptive et al.