That's folklore, not fact. The short form of the argument is: the studies are old, not very credible, and based on a notion of "productivity" which is most generously described as vague.
See http://leanpub.com/leprechauns (disclosure: I'm the author) for the long form, with a detailed bibliographical investigation of the "10x studies".
The rest of the "truths" in the OP are of a similar caliber: they are more properly called "opinions". A better title would have been "Some opinions about programming that you don't necessarily share". I happen to agree with some of them; but that doesn't make them true.
Furthermore, the CHAOS report defines "success" as "shipped on time and on budget" without regard to profitability or anything else people here on HN know is involved in real success.
As a result, when the CHAOS report says 51% of [surveyed participants self-report that] projects "fail[ed] in a critical aspect," what they really mean is that 51% of projects didn't ship exactly on time and on budget. If a project shipped a month late and made $10 million, it's still considered unsuccessful.
Horse pucky.
That being said, I do believe the spirit of these kind of statements to be accurate. Some programmers seem to be constantly inspired, always surprising you with a solid, clean and simple solution. Others seem to get bogged down in complexity and create fragile solutions with bugs that continue to pop up for years. The latter needs to be nudged in the right direction regularly. If left to design anything that is too big on their own without guidance they can and will cause disasters.
If you have ever had to manage programmers, you know that there are one or two guys who you give the really complicated problems - and then you don't have to stress because you know it will be perfect - even better than perfect. The other group you have to be more selective with their tasks and plan to check in more frequently. I spend a fair amount of my time trying to figure out why one struggles so much and the other does not - and how to help the one who struggles to become better. I don't consider myself a master, but I am one of those people for whom it seems to come easily. I'm not sure if it's experience or just a certain type of mind? I just know that I've observed it many times at many different companies.
Good programmers create features, designs, and implementations which are more elegant, cleaner, more understandable, more robust, less buggy, and much more valuable to the end user.
Consider J. Average Coder. He implements a feature in a typically half-assed way. The feature isn't well thought out, it barely works from a user perspective and has horrible usability. Anyone who doesn't think this is how average coders work has never experienced software developed by average coders. More than that, it's buggy and hacky and it serves as a development time sink, a tar trap for the entire development team. Bugs get reported and because of how messy and convoluted and fragile the code is it takes a lot longer to fix each bug. And in the end the effective productivity of that coder ends up being astoundingly low. Because the end result is a single feature that has taken a tremendous amount of coding and especially bug fixing on his part and on the part of his co-workers. The ratio of total end-user value to total invested dev-hours is incredibly low.
Now compare this with Dudley Code-right. He comes up with not only an excellent feature with high-user impact but he also comes up with a very elegant way to implement it. Boom, it's done, and nearly bug free. What bugs are found are quick to fix because the code is so clean and well put together. The ratio of total end-user value to total invested dev-hours is through the roof in comparison. Not just a factor of 100 different, but factors of thousands different.
There are tons of easy examples out there in the real world. Look at open source projects with tiny dev. teams, like nginx or varnish, and the value of those code bases relative to the effort behind them, then compare that to some random line of business software abomination from the enterprise trenches that has had hundreds of thousands of dev-hours dumped into it.
It's not the same thing as what we call "evidence", which consists of actually going out in the world and collecting information from more than a tiny and highly biased sample (which, like it or not, is a fair description of the set of programmers most of us will run into over the course of a career).
For instance, for one "Dudley Code-right" who is just as you describe, how many "Dudley Code-fast" exist, who are just as you describe except that no one else can understand their elegant constructions and it turns out that the "nearly bug-free" part, well... disappoints? Dudley Code-fasts look highly "productive" for a little while, then cause big problems down the road.
What we're dealing with here is the representativeness heuristic, which often makes us misjudge statistical truths. It's akin to the way people are much more afraid of flying than of driving, even though the former is orders of magnitude safer.
Now compare this with J. Average Coder. He implements an excellent feature with high user impact in a typically half-assed way. The ratio of total end-user value to total invested dev-hours is through the roof in comparison. Not just a factor of 100 different, but factors of thousands different.
So even if you were to accept that some programmers are orders of magnitude better than others, at it's likely that some of that is about things like noise, interruptions, network effects of putting a lot of good people together, and general corporate culture stuff.
My thought is that programming sees similar dynamics as the rest of the world. There are amazing people out there who build companies or change governments or move the world forward in a way that is beyond most individuals' mental capacity. Why would engineering be any different?
It is strange that the idea that "A good programmer is ten times more productive than an average programmer" is almost always taken out of the context of the human race as a whole.
I think it really comes down to the ability to find a problem and invent a solution, a rare gift indeed.
See also:
http://news.ycombinator.com/item?id=4118034
http://www.reddit.com/r/programming/comments/n8fqi/there_is_...
Forget about the studies. We all have seen empirical evidence of this.
I only relatively recently grokked this, and it's helped me to stick to best practices and naming conventions when in the past, I'd just throw in the first thing off the top of my head.
Conversely, I have to do something like read or watch a movie before bed to get my mind off of whatever project I'm working on or else I will lay awake thinking about it.
These generalizations seem too high level to be of any use, anyway. Good programmers want to program, and their interest will carry them up through skill levels. They'll want to learn and improve, and if they can't learn from your workplace you're wasting their time and yours. Bad programmers don't want to program, and should do something else.
This hasn't been the case on any non-trivial project I've been involved in.
However, a defining quality of successful projects I've been involved in is that there is one person driving the overall design and that person runs things closer to a dictatorship than a democracy.
Yeah, that's been my experience too, but I keep running into people who think everything runs better if the whole team decides and everyone gets a say. OK, there's definitely merit in getting input from what one hopes is a bunch of smart people, and it also helps morale for everyone to feel like they're being heard, but at the end of the day I think there really needs to be one or at most a few people who make the final decisions.
