> Right now we don’t know the answer to those questions.
I know the answer to the first three. Yes, yes, and yes. I've done them all, including all of them in the past few weeks.
(Which is how I learned that it's much better to ask ChatGPT to use Python evaluation mode and Pandoc and make you a PPTX, than trying to do anything with "Office 365 Copilot" in PowerPoint...)
As for the fourth question - well, ChatGPT can give you better advice than most advice on management/leadership articles, so I presume the answer here is "Yes" too - but I didn't verify it in practice.
> current systems aren’t creating real enterprise value at this moment in time
Yes, they are. They would be creating even more value if not for the copyright and exports uncertainty, which significantly slows enterprise adoption.
You say this but from a management perspective at a large enterprise software company I have not seen it.
Some of our developers use copilot and gpt and some don't and it is incredibly difficult to see any performance difference between the groups.
We aren't seeing higher overall levels of productivity.
We aren't seeing the developers who start using copilot/gpt rush ahead of their peers.
We aren't seeing any ability to cut back on developer spend.
We aren't seeing anything positive yet and many developers have been using copilot/gpt for >1 year.
In my opinion we are just regaining some of the economic value we lost when Google Search started degrading 5-10 years ago.
You can't measure productivity for shit, otherwise companies would look entirely differently. Starting from me not having to do my own finances or event planning or hundred other things that are not my job description, not my specialty, and which were done by dedicated staff just a few decades ago, before tech "improved office productivity".
> We aren't seeing the developers who start using copilot/gpt rush ahead of their peers.
That's because individual productivity is usually constrained by team productivity. Devs rushing ahead of their teammates makes the team dysfunctional.
> We aren't seeing any ability to cut back on developer spend.
Devs aren't stupid. They're not going to give you an opportunity if they can avoid it.
> We aren't seeing anything positive yet and many developers have been using copilot/gpt for >1 year.
My belief is that's because you aren't measuring the right things. But then, no one is. This is a problem well-known to be unsolved.
As a dev, you can use the saved time to slow down and not be stressed, spend more time chatting with colleagues, learn new skills, maybe improve the quality of the code, etc. Or you can pass it on to management which will result in your workload being increased back to where you are stressed again and your slower colleagues will be let go, so now you get to feel bad about that and they won't be around to chat with.
I have never in my life seen workers actually get rewarded with pay raises for improved productivity, that is just a myth the foolish chase, like the pot of gold at the end of the rainbow.
I have also tried being the top performer on a team before (using automation tools to achieve it), and all I got was praise from management. That's nice, but I can't pay for my holidays with praise, so not worth it.
For example:
- time between PRs being created and being picked up for review and merged
- time spent on releasing at end of sprint cycles
- time spent waiting for QA to review and approve
- extreme scrum practices like "you can only work on things in the sprint, even if all work is done"
How are you measuring developer productivity? Were those that adopted copilot and chatgpt now enabled to finally keep up with their faster peers (as opposed to outstrip them)? Is developer satisfaction improved, and therefore retention?
I guess we will see if smaller startups without many of our bottlenecks are suddenly able to be much more competitive.
> How are you measuring developer productivity?
We use a host of quantitative and qualitative measures. None of them show any positive improvements. These include the basics like roadmap reviews, demo sessions, feature cycle time, etc as well as fairly comprehensive business metrics.
In some teams every developer is using copilot and yet we can't see any correlation with it and improved business metrics.
At the same time we can measure the impact from changing the label on a button on our UI on these business metrics.
> Were those that adopted copilot and chatgpt now enabled to finally keep up with their faster peers
No.
> Is developer satisfaction improved, and therefore retention?
No.
Those are very high level. If there's no movement on those, I'd guess there are other things bottlenecking the teams. They can code as fast as possible and things still move at the same pace overall. Nice thing to know.
If you want to really test the hypothesis that Copilot and ChatGPT have no impact on coding speed, look at more granular metrics to do with just coding. The average time from the moment a developer picks up a work item to the time it gets merged (assuming code reviews happen in a timely fashion). Hopefully you have historical pre-AI data on that metric to compare to.
Edit: and average number of defects discovered from that work after merge
> We aren't seeing the developers who start using copilot/gpt rush ahead of their peers.
You think we are antsy worker bees, hastily rushing forwards to please the decision maker with his fancy car?
You are leadership. It's not hard. Cui bono, follow the money, etc. The incentives are clear.
If me and my peers were to receive a magic "do all my work for me" device I can assure you exactly zero percent of that knowledge will reach your position. Why would it? The company will give me a pat on the back. I cannot pay with pats on the back. Your Tesla cannot be financed with pats on the back. Surely you understand the nature of this issue.
Can you elaborate on what this saved over just making the ppt the old fashioned way?
Attach notes, paste, press Enter, wait half a minute, get back a PPTX you can build on, or just restyle[0].
Sure, it's faster to build the presentation yourself than to make ChatGPT make the whole thing for you. But the more time-consuming and boring parts, like making tables and diagrams and summaries from external data or notes, is something ChatGPT can do in a fraction of time, and can output directly into PPTX via Pandoc.
(There's a lot of fun things you can do with official ChatGPT and Python integration. The other day I made it design, write and train a multi-layer perceptron for playing tic-tac-toe, because why waste my own GPU-seconds :).)
--
[0] - In contrast, if you make the same request in PowerPoint's O365 Copilot, it'll barf. Last time I tried, it argued it has no capability to edit the document; the time before that, it made a new slide with text saying literally "data from the previous message".
The problem is that at the average medium sized company code looks like this - you have 1mln lines of code written over a decade by a few hundred people. A big portion of the code is redundant, some of it is incomplete, much of it is undocumented. Different companies have different coding styles, different testing approaches, different development dynamics. ChatGPT does not appreciate this context.
Excel has some similar problems. First of all Excel is 2 dimensional and LLMs really don’t think in 2 dimensions well. So you need to flatten the excel file for the LLM. A common approach to do this with LLMs is using pandas and then using the column and row names to index into the excel.
Unfortunately, excels at companies cannot be easily read using pandas. They are illogically structured, have tons of hardcoding, intersheet referencing is weird circular ways and so on. I spent some time in finance and sell side equity research models are written by highly trained financial analysts and are substantially better organized than the average excel model at a company. Even this subset of real world models is far from suitable for a direct pandas interpretation. Parsing sell side models requires a delicate and complex interpretation before being fed into an LLM.
