Seems we're not really disagreeing about the reality. We do seem to have different... interpretations, maybe?
"The second item specifically highlights documentation cost overhead of traditional methods as a major grievance"
Comprehensive documentation may be a major grievance, but it is pales in actual impact on the organisation, team or project when compared with the extortionate cost of discovering a design error during the test phase.
"Are we talking about different things when we talk about design?"
Two answers. First one is yes, I think we're talking about the same thing. Design in the sense that I use it is system design, or architecture. And yes, in a waterfall world that output typically is documentation + diagrams.
Second answer? Design how I do it doesn't involve documentation at epic war-and-peace proportions. A diagram can be very useful when communicating complex designs to external (to the team) stakeholders. A one-pager with VLAN descriptions and IP addresses can be helpful to a pen tester. However, in my world it's about a plan -
- What's the availability target, and how will it be met?
- Is the database relational, document-based, or are we using a lambda architecture? Justification?
- What's the dev plan? I.e. what feature dependencies exist, and how will they be prioritised? Consider for example a client app that communicates with a USB device. Initially a simple web site might be enough, but ultimately you need to choose between a native app; a daemon that bridges USB and browser via HTML websocks, or a daemon that bridges USB and browser via web site.
"although I don't view the waterfall process as being predictable"
It is extremely predictable in that phases are sequential, and you know exactly which skills are required for each phase. You'd use a business analyst to elicit requirements, and you'd not use a tester to write code. That means you can schedule resources across projects horizontally, instead of leaving them on one project until complete.
You mention "in any meaningful sense", and you're very, very correct to include that. The point is that waterfall is just as efficient at delivering quality code quickly as any modern agile methodology is. What's required is an experienced team.
Consider that an inexperienced team will screw up on an agile project just as much as they would on a waterfall project. Here's an analogy (taken from a blog post I'm still working on):
--
Karate is a martial art. It is also a competitive sport, and has strict rules. Krav Maga is street fighting. Anything goes - no rules, no competitions. I've been doing Krav Maga for over four years, and think it's the best self defence system known to man.
Saying that Krav Maga is the best is interesting. Not because it really is the best, but because that statement demonstrates two important things:
- It demonstrates consistency - my desire to be (and appear to be) consistent with what I have already done. Once we've made a choice, we encounter personal and interpersonal pressures to behave consistently with that commitment.
- It ignores reality - my girlfriend, a 3rd Dan black belt, has been doing Karate for over 18 years. Whether Krav Maga is better than Karate on not is immaterial - she'd kick my arse from here into next week. The reality is that she's simply more experienced. No matter how brutal a counter attack in Krav Maga is, she's so fast that there's no way I can even block her attack (I know because we spar).
--
It's that concept of experience that I refer to when talking about devs/hackers/programmers growing up.
As devs/hackers/programmers I think it serves us well to understand both extremes (waterfall and agile), and move the slider between the two as required. The mix should be appropriate for the team's experience. Enough prescriptiveness from waterfall to coach where coaching is needed, and enough agile to ensure that mistakes don't become expensive. Common sense should not be ignored when dealing with documentation, TDD, pair programming and the like, and nobody should ever be elected scrum master just because they paid someone for a piece of paper saying they can do it.