The problem I see is that people aren't really willing to be honest about the cultural transition that happens, so we also can't be honest about the process transition.
I think Agile-ish approaches work very well in startups, because the structure is pretty flat, and the goals are shared. But as companies grow, they tend to become what I think of business feudalism: hierarchical, control-oriented, territorial. For that, it makes sense you need different processes. And I think large company Agile is in effect Waterfall with a faster cadence, so you get that different process. But nobody will admit it. "We're doing Agile," they say, with too-bright eyes and gritted teeth.
What I wonder is: what if instead of killing the peer culture and the human-centered process as we scaled, we kept them?
I think that the big challenge is to recognize that hierarchical structure isn't good or bad, it's something that most natural systems do in response to scaling. If we understand it as a design parameter we can make it humane.
When you talk about hierarchy as natural, I'm not sure we're talking about the same thing. I mean hierarchy as a system of power/control relationships, with each unit having an actor in charge and totally containing sub-units. E.g., the classic army chain of command. I can't think of anything natural that has that. Flocks and troops have leaders, but I think humanity is unique in its extension of social dominance to many-layered nested structures controlling millions of individuals. So I'd call it essentially artificial.
I agree that we can apply a hierarchical analytical model to many natural systems, but I think there are plenty of other models we could apply, just ones that aren't as easy for current human brains. E.g., we could talk about fractal, self-similar, or recursive processes; those have no connotation of primate dominance. Or note the shift away from "great chain of being" analyses of nature toward ecological and non-teleological analytical models. When all you have is social primates, everything looks like a hierarchy.
I would certainly like to believe that we can make hierarchies humane. But given that a fundamental characteristic is individual power over others that increases as you go up the chain, and given human fallibility, cognitive limits, and human nature, it seems to me they have to become essentially inhumane at some size. And in practice I certainly see a correlation between hierarchy size and inhumanity.
Correlation isn't proof, of possibility, of course. Just natural tendency. Are there large hierarchies you think violate that correlation, where they have gotten more humane with scale?
When administering computers we talk about treating them as "cattle, not pets" to increase our ability to manage large systems. Using the same metaphor with people would be offensive but the policy-making impulse is similar.
To some extent it even arises for valuing fairness. We make rules to try to treat people consistently and fairly, forgetting that people are a mess of special cases.
Small companies value consistency too (using the same tools and procedures) but it's just easier and less likely to backfire with fewer people.
In which case, I'd like to see what happened if we favored humanity over scale.
If you pay attention, the best Agile systems focus on improving the skills of developers instead of forcing people to follow the 'steps' or a formula.
Of course a design that works at 1x doesn't always work a 10000x, but why shouldn't a design that works a 10000x not also work at 1x?
At least for technical scalability the answer almost always lies in overhead. Yes, while the system scales perfectly linearly at O(n), the hidden constant is often neglected, which means at small values of n the solution becomes unfeasible. So while Cassandra might for one of my use cases be the perfect candidate due to its feature set and linear scalability it becomes unfeasible at a small scale because I have to run at least 3 nodes with high overhead (because Java, sigh) + an additional 3 nodes with Zookeeper with high overhead (because Exhibitor + Curator, all in Java).
/rant, I don't like bashing on Java but overhead is definitely one of its weak spots, and here is where it comes into play.
So, I don't think we can take it as far as author is suggesting where it's like classical vs quantum physics. Maybe at ASIC vs software level. Even those were partly joined with synthesis & coprocessing tools. I just don't see it as every high-level description I read about things is based on similar principles at each layer of the stack. Given similar constraints or goals, you would use similar strategy. There's certainly divergence or outliers but more repetition of patterns than anything.
The only myths are that technology/fads X, Y, and Z should've been widespread adopted over the ones (or enhancements of them) that consistently worked. The author is seeing results of people building on stuff that came with assumptions that don't match new problem. Or people straight-up ignoring root cause in their solutions or methods. Common problems.
What about starting off with microservices (because tools like k8s make the otherwise insane management overhead more tractable)? Could that be a possible solution?
The problem of scaling state, databases need to handle anyway. Ideally the same database just keeps on working as you go up. Or atleast the protocol is the same, and you switch out the implementation from a single instance to a clustered version.
So, every process around being able to deliver that by just applying that process creates a false hope. Or it is not meant for this purpose but the "buyers" aren't aware of it and have other expectations.
I'd like to give you a simplistic answer, like "All you need, kid, is a small team! For anything!"
Slogans like that are true, but yet they are terribly misleading, because 1) many organizations are already terribly overstaffed, and 2) it doesn't really help to tell teams "What do I do right now?"
So here's as simple as I can make it:
Good organizations will do whatever is necessary to make things that people want, even if that means instead of programming, the programmers sit on the phone and do some manual chore for people as they call. Before you code anything, you have to be hip-to-hip with some real person that you're providing value to.
But as soon as you have those five folks sitting on the phones doing something useful? You gotta immediately automate. Everything. This means you're going to have all freaking kinds of problems as you move from helping ten people a day to helping a million. You have to automate solutions, access, coordination, resource allocation, failovers, and so on -- the list is almost endless (but not quite)
As they grow, poor organizations take a scaling problem and assign it to a person. Somebody does it once, then they're stuck with it for life. Good organizations continue to do it manually, then immediately automate. Somebody does it once, then the team immediately starts "plugging all the holes" and fixing the edge cases so that nobody ever has to manually be responsible for that again.
Growing "sloppy" means you end up helping those million people a day -- but you have hundreds of people on staff. Meetings take time. Coordination takes time. There are a ton of ways to screw up. People tend to blame one another. Growing good means you can be like WhatsApp, servicing a billion users with a tiny team of folks.
If you're already an existing BigCorp and have been around for a while -- odds are that you are living with this sloppy growth pattern. That means you need to start, right now, with identifying all the feedback loops, like release management, and automating them, such as putting in a CI/CD pipeline. Not only that, but there are scores of things just like that. You have a lot of work to do. It might be easier just to start over. In fact, in most cases the wisest thing to do is start over.
Now picture this: you're an Agile team at BigCorp and you've got the fever. Woohoo! Let's release often, make things people want, and help make the world a better place. But looking around, all you see is ten thousand other developers in a huge, creaky machine that's leaking like a sieve. You go to a conference with a thousand other BigCorps, just like yours. Are you going to want to hear about how it's better just to trash things and start over, about the 40 things you need to have automated right now but don't, or how to make your section of 150 programmers work together; how to "scale agile"?
Scaling Agile is an issue because the market says it is an issue.
