Gottfried Leibniz suffered no small amount of flak for his conclusion that the reason the world looks the way it does is because it's already "the best of all possible worlds": you may think you see an improvement, but you lack divine understanding and therefore you don't see how it would actually make things worse.
While this is in my opinion a silly idea, I think most people who work in tech could use a bit of humility influenced by this line of thought: if a solution exists today, and it has yet to be replaced, it's at least possible that it's over all the best solution, and the reason you're not seeing that is because of a lack of understanding.
Of course this is not always true, but if nothing else it would lead to more interesting discussions than those stemming from someone saying "if you're not using Kubernetes you're a moron" and another replying "learn how to sysadmin and you'll realize you don't need Kubernetes in the first place".
We should IMO also entertain one more possibility: namely pressure from the top.
A lot of people who I knew as sympathetic and calm before they took management roles turned into something I could code in one minute: namely a program that asks "how much is this going to take?" and if your answer is above N hours/days then they say "no, we're not doing it". And that's not because they are stupid or suddenly non-sympathetic. It's because their bosses optimize for those metrics and measure them (and their salary, and their bonuses, and even their vacation days) by the same metrics.
Perverted incentives play a big part in the current state of things where everyone just looks baffled at thing X and is like "why in the holy frak isn't this working better?".
Truth is, MANY, and I mean really A LOT of people know that their work is sub-optimal but pressure from above prevents them from making it better.
And people are not independent -- for most people outside IT and other prestige / high-skill professions, standing up to the boss means 3 to 9 months of unemployment and maybe no food on the table. Hence everybody learns to suck it up and do the bare minimum that keeps them employed, perpetuating the cycle.
Multiply that to at least 90% of our world and it's IMO quite easy to understand why it is like it is.
But I'd argue that most 'computer nerds' have pretty deeply seated perfectionism flaws.
To take a simplified example, we tend to want perfect security, and like to prove that we're smarter than each other by understanding complicated attacks.
When it comes to running a business, though, a 10 person startup cannot afford to act like their threat model includes defending from hostile nation state actors willing to do Stuxnet-levels of effort to hack them. If they do that much work on security their product will never get off the ground.
The goal of most companies when it comes to security is never to have perfect security, but to have good enough security to stay out of the headlines.
It is like tradeoffs that have evolved in nature. You don't see organisms that maximize one feature at the expense of everything else. That energy always needs to go into competing concerns.
So sometimes as technical people we fetishize the perfect solution to some problem in the Enterprise, when actually doing that solution would be horrible for the Enterprise due to the costs.
(Of course the flip side is sometimes management only cares about window dressing the company so they can bailout with some golden parachutes and that has nothing to do with running a healthy company -- but not every example of a company not being perfect is due to deeply cynical and perverse incentives like that, but may be entirely rational).
In my point of view, that's how you lose solutions to stay in the comfort zone, and I'm not saying the people whom actually "just obey orders" but those who make them.
As I gain more experience, with age and some wisdom (I hope), I see that most top-down decisions are actually the worst way to decide things, people who are "at the bottom" maybe not understand how a entire system works however they do really knows (and suffer) on how their work is sub-optimal.
I had this exact same thought recently when reflecting on my behavior in my new role as a "technical product owner". All of it was reflexive, as if I suddenly forgot all of the software engineering knowledge I accumulated over the years and became a deadline-driven cog.
I don't have a solution yet; I think it comes down to that I don't yet speak the same language that people I report to do, and thus I feel like I can't defend my position well enough. It comes with experience, I guess!
We as an industry tried the all engineers thing and it didn’t work.
Have you considered the possibility that the boss’ bosses have a point? That the intelligent people you once respected have been convinced rather than intimidated?
> if a solution exists today, and it has yet to be replaced, it's at least possible that it's over all the best solution, and the reason you're not seeing that is because of a lack of understanding.
And once you build mastery of $fwk/$lib/$tool, it's deprecated and you migrate to another one. GOTO START
Management and owners are a far from perfect group that make plenty of bad strategic decisions for a host of reasons, but they also have to deal with the real/perceived constraints of time, budget, market, competition, etc. that are easy to not appreciate when you are focused on the design/build portion of the business.
Then you look a little deeper and realise most of them are pretending, they're photoshopping their selfies, renting their expensive cars for a day and taking their selfies on other peoples property.
HN can be a bit like that, you can feel completely inadequate compared to these super-geniuses doing everything perfectly, when really you're doing ok.
HN attracts a broad range of ability, but generally upvotes the smartest content. So if you sample from “upvoted HN articles and comments”, the content you read is heavily biased toward things written by unusually smart people. But there’s a range.
By contrast, most companies will attract and hire people like their current employees. So the variance at any particular company is much smaller than in the general pool. Everyone ends up working alongside people like them.
So if you work at a big bank or something, you probably don’t interact with many people like the super geniuses who write interesting articles that get upvoted a lot on HN. They’re over represented amongst HN content, but most programmers are average and getting by just fine. And remember, hotshots are equally ungrounded. Lots of good engineers you meet at Google were hired at straight out of collage. They have no idea how the sausage is made anywhere else. Lots of folks here don’t have your experience. They have no idea what it takes to do software at a company that sees their job as a cost centre. Or what it takes to get good work done in spite of political pressure to stagnate.
Having a variety of voices gives all of us something to learn and something to teach.
The more complex solution is to develop a worldview where you only compete against yourself, i.e. is what I'm doing now going to make me a better person in X time.
It's up to each person to decide for themselves what better means, it could be an emotional thing "will it make me more empathetic?" or a financial thing "will this allow me to earn more money?" or something else "will it make me a better programmer, chess player, cyclist, *happier*?".
For all the size of the world and its complexity it collapses down to a local optimisation problem for you as a person.
Even if everyone were to post the mundane it wouldn't generally be what would get upvoted or liked in a given medium enough to counter the successes.
However we are humans, only recently evolve from something a bit like an ape. Hampered by emotions and constant fits of irrationality. Actually achieving the correct behaviors that we can easily imagine is very hard and very rare.
So we are correct in our criticisms but could be more calm and understanding about the fact that everything is terrible, and strive to make even very small improvements and enjoy those.
I don't believe this is true, an average person can easily imagine a local maxima, but everyone's local maxima is different.
For example it might be very convenient for person A if the licensing office were open 24 hours but probably not so convenient for person B who would have to cover the night shift every other week or person C whose taxes would need to pay for it. And so on.
Nearly every sub-optimality from one perspective is an optimality from another.
I'm willing to assume that this is the best of all possible worlds, that everybody did the best they could in the moment. But having all that history behind us gives us lessons such that tomorrow's best possible world is surely better than today's.
I agree that we should have less technical arrogance and jackassery in the future. But that emotional immaturity is also part of today's best possible world.
[1] https://en.m.wikipedia.org/wiki/Wikipedia:Chesterton's_fence
- Usually, the "right way" to do something is only apparent halfway through doing that thing, or after you finished that thing.
- There's rarely time to redo things with benefit of hindsight, because the priority is always to work on business goals first.
That's how things were everywhere I worked or seen, so I believe this is likely a general phenomenon. But the consequence of that is strong path dependence. It's not that using Kubernetes makes you a moron or a smart person. It's that somebody already deployed Kubernetes for a project that's too simple for it (or they were too inexperienced), but business pressure makes it impossible to spend a couple of weeks undoing it, so everyone is stuck with the pain. Perhaps the reason for the initial Kubernetes deployment was because it was the fastest way to fix some other issue, and the company could not allocate enough time for engineers to do it right.
To be clear, this isn't me ranting about "stupid managers with their priorities". Pursuing business goals is the raison d'être of a software company (it's pretty much a tautology). I'm just saying, it's hard to improve things without lots of slack, and slack is a competitive disadvantage.
Perhaps another facet to this is: business lifecycle is too short for software lifecycle. If you were to spend time on redoing things properly, by the time you were done, you'd be out of business.
Think of it like gradient descent. The location you end up is the best location in the neighborhood of your starting position, but it's almost certainly a local minimum, and if you started over from a different initial position there's a good chance you'd find a lower minimum.
Since we can't avoid thinking in generalizations, and they're usually false, the only option is to accept contingent thinking. Sometimes, respecting "the way things are" is the right way. Sometimes, rejecting the way things are is the right way. It's often how things get better. There isn't a true "way" you can live by in all circumstances.
There are hints though. If you're using disrespect/challenge to write a good essay (like this author) or to try and change something.... rejecting "the way things are" might be good. If you're just whining, probably not.
Humility good. Complacency bad. These are definitions, not value statements. They're not necessarily contradictory, if you care about such things... but they can have a Hillel/Shamai^ dynamic to them. Sometimes, an adversarial discussion like "Kubernetes" vs "Learn Tow to Sysadmin, You Moron" is enlightening. Sometimes it's no-it-all asholery. I don't think there are universal rules governing the distinction.
In any case, I don't think this author is being "terrible." He does recognize that scaling is a problem that he doesn't fully understand and meanders through a lot of possible reasons. None of it is mean. The title is a little click-bait, but I don't think it's a foul. He's quite respectful, in fact.
^Like political left/right, but more bearded.
It's like applying principles like DRY. If you push that principle to the extreme, your code will be shit. DRY is a good principle, but not a rule to follow blindly.
Basically he's investigating and goes and talks to the software guys and quickly concludes everything is awesome, and moves onto investigating hardware.
I was initially taken aback by this part. But he later he wrote:
"To summarize, then, the computer software checking system is of highest quality. There appears to be no process of gradually fooling oneself while degrading standards, the process so characteristic of the solid rocket booster and space shuttle main engine safety systems."
It makes sense, I mean they can fly simulations over and over before the shuttle is even built, let alone launched.
[1] The first book was "Surely You're Joking Mr Feynman", the second with the challenger investigation was, "What do you care what other people think?"
Large systems in general are hard to refactor. Our impulse in tech is to always consider the refactor because we see how in other systems a refactor is impossible or takes blood (our governments and countries, institutions like education/healthcare). Those in tech have seen many systems to compare against (what went wrong, what went good, even on the most basic projects).
The first thing they tell you about capitalism is that humans haven’t encountered a better system, which is true. Make that argument to a programmer. They either saw a shittier system, a better one, a more fitting one, a proper one, a terrible one, etc. You pretty much can’t tell us anything because we’ve seen it somewhere.
Being content with systems is bad for all of us. Being content with your life is good for all of us.
In my opinion, what makes most of the wrongs we see is that computing is very young and we haven't had the time to set up half decent solutions for anything.
There is a huge demand for any solution, because a bad solution is better than no solution. The Kubernetes vs sysadmin is like the Diogenes story. There are two versions. In one of them someone tells Diogenes "if you praised the king, you wouldn't need to eat lentils" and he replied "if you ate lentils you wouldn't need to praise the king". In the other version it's just the same story only Diogenes talks first.
I don't like lentils. I don't like to praise the king either. Both are terrible solutions, but while we have kingpraiseless burgers, we need to choose something that somehow works.
It also doesn’t have to be the best, and probably isn’t – the space of all possible bit strings large enough to encode Kubernetes is quite big. Being “good enough” is sufficient, and often the discussion really comes down to differing opinions on what that means. These is no uniquely and clearly defined objective function here. People need to stop thinking that theirs is the one and only.
Precisely. And, building the truly best solution, is often a mistake, in that it can take pointlessly long.
Good enough is good enough, and others might find things to improve, and be correct that it'd been better in their way — the important thing is instead to realize that actually making any improvement one might have found, can be far down on the priority list. Maybe there'll always be other more important things to do, so it'll never happen.
> > "at least possible that it's over all the best solution"
can be rewritten as:
> > "at least possible that it's a good enough solution"
And maybe it can make sense to make improvement suggestions, and then, when the others say "yea but there's too much else to do", then, ok, maybe forget about it and be happy anyway.
I've been on both sides of this, and look back and cringe when I think about projects I've criticized before trying to understand them. I've also had the experience of being early at a startup and having later employees come in and constantly complain about how sub optimal everything is. The problem is they're missing the time crunch that was in place, and the fact that if subpar solutions weren't used the company would have never gotten to market and they would have never gotten a job there. Software isn't created in a vaccuum.
Even the best possible solution is only best in the time it was created, and because it's good it will get entrenched and keep getting used, even when the constraints it was created in no longer apply even remotely and it's actually a very suboptimal solution given new constraints.
Everything from programming languages, OS kernels, file systems, distributed computing frameworks - soo much of it has ideas and solutions that are technically better but it's hard to overcome the inertia
I'm sat in front of a machine that is literally billions of times faster and many orders of magnitude more powerful in every other axis (storage, bandwidth).
And yet, strip away the fancy IDE, the container tooling and it's all still my brain, some text on a screen and my understanding of the problem.
https://en.m.wikipedia.org/wiki/Pareto_efficiency
https://en.m.wikipedia.org/wiki/There_ain't_no_such_thing_as...
Either way might be a better choice for a given situation and due to lack of information and changing environments carry their own risks.
If you don't know what these reasons are, it is hard to know with certainty that they no longer apply.
see above reference to Chesterton's fence.
Isn't this line of thought disproven by a single improvement? People before the improvement could have argued along the same lines and then be proven wrong.
Maybe the field of software can really mature only when looking for subtractive solutions become part of the computing culture.
Agree. See: https://fs.blog/2020/03/chestertons-fence/
That said, we may not get far if no one makes an attempt to tear down those fences. Tech is like art in that respect: Lets practitioners experiment like no other engineering discipline.
Ideally you want 'seasoned vets' for whom this level of engineering/development is old-hat and have them churn out a solid product that handles all the edge cases that you only know about when you have a long-term experience with the technology/domain.
Still, looking back we can see how software solutions were not optimal for their age, and how amazing things can be coaxed out of old hardware using new insights. It would be pretty remarkable that precisely now is when we finally achieved optimal software. More likely, we’re still getting it wrong, and this is just one more step along the line.
The truth can be a little of both: yes, we can do better than what exists now, but we’re not guaranteed to unless we understand what came before.
The thing is, most software projects have an inherent complexity that can’t be reduced. This gives an implicit limit on what can be reasonably created by humans.
And most software is insanely complex — simply because otherwise they would not be useful in many cases. So writing software from scratch takes pretty much the same amount of time as it did 10 or 20 years ago. Yeah performance is much better both in hardware and in software (runtime, compiler), but eg. desktop app development may have become less productive since. All in all, there is No Silver Bullets, and there haven’t been since either. The only thing there is is incorporating already existing software.
There’s also the case where you or someone is hired to offer a fresh perspective on an existing system. After looking at it if your solution is “it looks great the way it is”. The person who hired you would think “why the hell did I even hire this guy?”
So even though you know the current system it’s a good system you try to propose a solution just to show value. That it was worth it to hire you.
This is true of any "real world" subset, vs its online outlets; HN may just be more niche than some.
K8s is just a container orchestrator. And containers are just virtual machines.
> containers are just virtual machines
I think this part is false though. Containers are, well, contained processes with contained storage. No simulated hardware like VM implies
People don't want solutions to yesterday's problems. These are considered trivial and already solved, such as invoking a shell command (which just hides a lot of complexity under the hood). Noone will pay you for invoking existing solutions. They pay you to push the boundary.
By tooling I mean programming languages, frameworks, libraries and operating systems. All of whuch have been designed for a single machine operation with random access memory model.
This no longer holds true. In order to scale software today you need multi machine operations and no such OS exists for it yet. Only immature attempts such as Kubernetes, whose ergonomics are far from the simolicity that you'd expect from a unix like system.
And random access memory model breaks down completely, because it is just a leaky abstraction. For full performance memory is only accessed linearly. Any random access structured programming language completely breaks down for working with GBs of data, in parallel and on a cluster of machines.
I don't think we'll push the boundary further if we keep piling crap on top of existing crap. The foundations have to change and for that we'd better go back to the 70s when these ideas were being explored.
Most businesses pay developers to solve problems, not to push boundaries. They typically prefer developers using "boring" tried-and-true approaches.
In my experience it is developers who push for "boundary pushing" solutions because it is more exciting. And of course the vendors selling those solutions.
I am not exactly sure what you mean by this, but taken literally; most companies pay well for this and it's also the most common work programmers do all over the world. But maybe I misunderstand your meaning.
Why would you need a cluster of machines to work on mere gigabytes? You can get single-processor machines with terabytes of ram. Even laptops come with 64GB now.
I wouldn't put it like that. 90% of DevOps jobs is to invoke the existing solutions quickly. Sure, there will be a little bit of pushing the boundaries here and there was needed. But largely I'd describe those jobs as putting the Lego blocks together every day - and there's lots of money in it.
The other part of my job is more difficult, which is encouraging teams to de-silo and automate wherever possible but so many of the employees are threatened by this idea at Big Corp that I'm encountering a LOT of resistance.
We saved multiple customers a lot of time and money by just doing minor load balancer or web server configurations, rather than letting their developers doing a custom application to handle things like proxying or URL rewrites. Similarly we also frequently have to ask customers why they aren't offloading work to their databases and let it do sorting, selection, aggregation or search.
Having just a small amount of knowledge about what your existing tools can actually do for you, and knowing how to "Lego" them together solve a large number of everyday problems.
You'd be surprised. Most software production re-builds and re-invokes existing solutions that just aren't systematized yet. It certainly doesn't push any boundaries...
Meanwhile, there are things sitting on the shelf that solve these problems in a principled way and make things simpler:
- ML languages
- Nix
- Bazel
- Erlang
- Rust
Some tools that are great and have large adoption:
- Git
- Terraform
ML was first developed in 1973. Ocaml in 1996 and SML in 1997. Great tools which haven't been popular in 20-40 years probably have something beyond fear of early adoption inhibiting them.
Some companies are "lucky" and have either natural sharding keys for their primary business line, are an easily cacheable product, or can just scale reads on replicas. Others aren't, and that's where things get complicated.
Same goes for the other items in your list. Git had enough time and force behind it, and I believe the other tools will succeed as well. But it will take time.
Large software projects do not have decades. Only few years.
Distributed operating systems are nothing new though. Plan9 for example.
It is my firm belief that the vast majority of value in Software Development is in understanding the problem and the ways that Software can help. Crucially this includes knowing when it can't.
When asked what my job was once I said, "to think and write code, the hard part is doing it in that order."
That said, it's implausible that the main problem is scaling since mainframes scaled considerable data quite a while back.
I've been working on a new type of framework that is about making how we develop software match this mental model. Where the higher-level primitives we think in terms of are first-class citizens at the language level.
I think only then is it possible to reason about, and scale, software in a much more productive and effective way.
I view this a little differently. People have tried again and again to move the abstraction layer for coding above text files. This was happening back in the 90's (in the guise of so-called "4GLs" [0]) and still today (now rebranded as "no-code" [1]). I myself spent a good deal of effort trying to code "diagramatically" through top down UML (with an amazing product for it's time, Together). So the ambition to shift programming up the abstraction food chain has been tried continuously for 30 years and continued to fail every time. Eventually I changed my view and decided that there were fundamental reasons why higher level abstractions don't work - software is just too complex, the abstractions are too leaky. The details matter, they are critically important. We are in a long arc of figuring out the right abstractions that may take a century or more and in the meantime, we simply have to have the flexibility of text based formats to let us express the complexity we need to manage in the meantime.
[0] https://en.wikipedia.org/wiki/Fourth-generation_programming_...
[1] https://techcrunch.com/2020/10/26/the-no-code-generation-is-...
You should learn about Elixir (or anything on Erlang VM).
People are paid to work on products which provide value to the business. There is always a Minimum Viable Product, MVP. Meet that. Exceeding that is the boundary that needs to be pushed, not grabbing the latest, possibly untested, tools off the shelf.
> And random access memory model breaks down completely, because it is just a leaky abstraction. For full performance memory is only accessed linearly. Any random access structured programming language completely breaks down for working with GBs of data, in parallel and on a cluster of machines.
This is why constantly re-engineering (for the purpose of engineering) is not the most useful method. I used to work in FoxPro with databases that were GBs of data. If today those GBs are difficult to handle, which they aren't, then there is a problem with how the stack was put together. GBs are trivial.
We naturally organize into hierarchies of small subgroups with limited interaction boundaries. Each subgroup will adopt slightly different methodologies, have different tooling preferences, will have integration and communication overhead with others, and so on.
This can not be preventend, only mitigated. Which is perfectly fine. The important step is recognizing that software development is really not special [1], and that the limiting factor is the human element. Then you can build your organizational structures around these restrictions.
In the late '90s and early aughts everyone read Mythical Man-Month. There is even a law attributed to him, Brooks' law [1]. It really feels like no one internalized what he was actually saying. I suppose in a world of trends and fads, timeless advice is hard to recognize.
> [...] the increased communication overhead will consume an ever-increasing quantity of the calendar time available. When n people have to communicate among themselves, as n increases, their output decreases and when it becomes negative the project is delayed further with every person added.
Microservices, of course, exasperate this by actually increasing the communication required to make software.
I HATE microservices. I've never seen them used appropriately, although I imagine it can be done. But I think if you increase human communication by using microservices you're defeating the raison d'etre of microservices. That's not the fault of microservices - it's the fault of the humans using them badly or where they shouldn't.
Story time. I once contracted to a fortune 500 doing a $100M total rewrite with literally 100 microservices all in their own git repo. It was such a disaster I contemplated shorting their stock. I don't know how the story ended, but I feel confident it didn't end well.
Software companies/teams/applications of today are many time larger than in 1975. It may be more efficient to have software that's federated and developed by small teams but... Facebook's revenue is $90bn. A social network 10% FB's "size" will make much less than $9bn in ad revenues^.
That means that product teams grow until the point Fred Brooks warned about play out... until the marginal team members' contribution is negative.
^For example, Reddit has 10%-20% facebook's users, but only about 2% of its revenue.
I'd argue the exact opposite. Microservices with defined interfaces and APIs mean that everyone doesn't need to communicate with everyone. Only the people who work on a given microservice need to communicate amongst themselves. Across those groups the communication can be restricted to simply reading documented APIs. That in turn drastically lowers the amount of communication needed.
Scrum and other methodologies assume stable teams. It takes a few sprints for a team to assign somewhat accurate story points to stories. It takes time for people to find their roles in a team. Whenever someone joins or leaves a team, it takes time to find a new equilibrium. So if you aim for peak performance teams, it takes months of stability to reach that peak. You better not change it then or it will take months again.
In contrast, most organization have a constant need to reorganize to adapt to new business models, customers, product portfolio, etc. For consulting gigs which takes less than a few months, it makes no sense to aim for peak performance and stable teams. Instead you should aim for smooth transitions. In such an environment, more formal methodologies make sense because good documentation, unified code styles, and consistent processes make transitions more smooth.
Of course, this isn't a black and white decision but more of a scale from stability to flexibility. There are small lifestyle companies which are very close to the stability extreme (the Sqlite team maybe?). Valve with its flat hierarchy might be an example for the flexibility extreme.
1. The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale. In Coasian terms, the transaction costs around development are minimized.
2. It enforces hard interfaces and it clearly separates development from operations. Lack of discipline around these issues is a source of much accidental complexity.
[1] I can’t seem to find the quote, I read it in an interview a few years ago and stuck with me since.
If you grossly simplify it to its innermost core, making development scalable means that if you have 10 times more people you can have 10 times more features/bugfixes/speed improvements/... The only way to do that is to make sure that an additional developer doesn't depend on other developers to work, and that can only happen if everything is properly documented, the build instructions are up-to-date, the processes are clear, basically anyone can start from scratch and get up to speed without the help of anyone else.
That kind of organization traditionally doesn't happen in on-site companies, where newcomers are followed by senior people, they have to follow some introduction course to familiarize themselves with the processes, they have to ask many questions every day, they need to be synchronized with other persons which brings some inefficiency beacuse everyone works at their own pace, etc... This all disappears when everything is properly documentend and every contributor can work in the middle of the night if they wish. I think the Gitlab Handbook goes over this quite well and describes a framework to implement that kind of organization but the rules are retrospectively obvious for people already used to open-source (https://about.gitlab.com/company/culture/all-remote/guide/):
- write down everything
- discussion should happen asynchronously. Any synchronous discussion (by text or call) should be only very small points. Whatever the type, write down the conclusions of that discussions
- Everything is public (to the organization), including decisions taken, issues, processes
Another very relevant factor is that people can just clone stuff, create new projects, and everything moves independently. So the open source development model has a pile of solutions for dependency management that team based development doesn't adopt.
But the one thing I don't get is why team based development doesn't adopt those solution. They are not expensive. Yet, even when I was able to dictate into everybody's requirements, I wasn't able to force teams into adopting them. Instead they insisted on synchronizing themselves by much more expensive and less reliable means. My guess is that most developers never dug any deep into open source and have no idea how it's done.
“open source” isn’t a development methodology, or even a distinct set of methodologies.
> The project structure, tooling and documentation that lets new contributors jump in quickly makes software development easy to scale.
Plenty of open source projects don’t have that, nor is there anything restricting those things to open source projects.
It is true that some open source projects, because they see the value in new developers jumping in quickly, prioritize having structure, documentation, and tooling that supports that. It’s also true that some proprietary software projects do, too, because the project owner sees value in that.
Some companies such as Microsoft, Google, Amazon would disagree.
> Conversely, if we built our cities the way we build our software, you would need to enter the shop through the special garage, and exit through the roof to walk a wire to get to another custom made building from scrapped containers to do the checkout. And some of the windows are just painted on because they’re an MVP.
And then you have that one section left over from the original release that all the engineers agree desperately needs to be refactored and upgraded, but due to cost and politics they never get to do it. And anyway you have a couple of power users who insist that the backwards and broken way that part is implemented is actually perfect and shout very loudly anytime you suggest changing it.
Also like a sibling comment states, 4 developers is not even close to a "large" software project.
The problem with this industry seems to be that once something is solved, i.e. we already have reliable, battle-tested "streets", there is a big pressure to push everything further, build even more complex systems, faster. The pressure is rather natural: you will have a competitive advantage if you can push the limits and build something that can't be done based on the previous architectures, within a limited time frame.
For example, building desktop apps is a solved problem. These are you streets, these are your building blocks. But because it's a solved problem there's little money to be made out of it. The money lies somewhere on the edges of the map (e.g. SaaS) where there are still no roads and no general urbanization plan, and it's where the businesses tend to flock to. Hence the chaos, uncertainty and quality problems in most of innovative software.
I live 1.6km from my office. I drive 9.2km to the office. Cities aren't as straightforward as you might think.
Whereas if I design something in CAD, I send it to the printer, try it, improve it once, print again and never ever again think of it as long as it does it's job.
Of course these concerns matter more if you’re making a lot of these “widgets” but even in the case of a one off you’re going to have servicing costs and the occasional redesign/replacement/upgrade ...
It’s not that far out of line with software - widely used products go through tight highly iterative development cycles whereas one off solutions tend to be just “good enough” with bug fixing and the occasional feature request.
No other machine built by humans is that complex.
The race to the bottom incentivises profit over principle. Boards don’t care if something is built to scale if the competitor gets to market with a cheaper solution. Scaling is a day two problem.
I’m bearish on my future employment as someone who questions the motivations for profit driven development.
I naively thought the market would reward the best product, instead I see the cheapest product being rewarded. What happened to innovation and doing what’s best for society?
That's because you are looking at best from one angle and the market from another.
To the "market" the best may well be the cheapest thing that mostly does it's job/most of the time.
When I was young I used to think users cared about bugs and they do but they only care when the bugs are sufficient in number and degree to cross a threshold where they outweigh the utility of the software, That sort of 1 in 100 times it loses an hours work, well if the 99 times it saves me an hours work, shrug.
What we (as techies and I'm generalising wildly here) want to build is heirloom quality carpentry, what we get to actually build is franken-furniture from ikea packs that we hope has some some instructions and won't fall apart next week.
This observation seems both true and important.
Add in high levels of turnover and unless the specific person who learned the lesson is also on the project, there isn't a clear wrong way to prevent a wrong turn.
I am part of a new team at a startup and everyone on the new team is a relatively new hire. As far as I know, we haven't taken any meaningful learnings. Despite the company existing for years, we have done everything from scratch.
One of the metrics that I use address your point:
Number of Ex-Engineers (NOEE): This is the total number of unique engineers who have touched a code and have left the company as of the release date of the software system
Implications: This measure deals with knowledge transfer. If the employee(s) who worked on a piece of code leaves the company then there is a likelihood that the new person taking over might not be familiar with the design rationale, the reasoning behind certain bug fixes, and information about other stake holders in the code.
A large loss of team members affects the knowledge retention and thus quality.
[1] https://www.microsoft.com/en-us/research/wp-content/uploads/...
Also, the way to disseminate knowledge is by talking about problems. As in, if you manage to create culture where people do that, you will occasionally hear "team x tried that and had peoblems, lets ask them".
But cities ain't pretty. Dig the ground and nothing looks like what's on the map. To create a new building, you need to cut a lot of plumbing and re-route it somehow.
Stuff gets old and breaks all the time, and maintenance teams are on the ground 24/7 to fix it. NYT subway is the mainframe of the city. Look at the central steam heating that's still pumping water into high rises.
Sure, you can sell a shop, and people will be able to repurpose that space to suit their needs very efficiently. But isn't that what Docker does? Provide a cookie cutter space to run an app?
But in cities, there are places where trucks can't go, where you can't park, where unplanned work happens, where the sewer floods. That's when the city 'bugs', and people need to manually intervene to fix it...
Trying to find an ideal way of running software at scale is just as utopian as building the perfect city using 'super structures'. It's a dream of the 60s that's never going to happen.
100% agree.
Best boy basically means 'chief assistant', second in command kind of thing.
Grips build the structures which cameras and lights are hung on. They don't touch the electrics, gaffers and electricians do that: grips make the towers, stands, tracks: the physical stuff on which you put things.
Does anyone have first hand experience with this working really well? It seems sensible but in my experience it does not work. The platform teams, I think, should be collaborations between a small number of members from each end-to-end team. The platform would be allowed to grow only as needed by the use cases. As soon as you put a platform team in charge with a mandate, what they build and what is needed starts diverging in the name of "Long term planning". Instead of fixing the struggles of today, they think they have the capacity and formula for fixing the problems of a year from now. In my experience, they do not. Would love to hear from others
From Jan Bosch, I heared the proposal to have "component owners" in addition to "feature teams". They are senior developers, normal members of feature teams, and explicitly responsible for specific components. This means they should review all code changes from any team but not do any functional changes themselves (as long as they wear the "component owner" hat), just maintenance. This might work. (Unfortunately, he hasn't published this idea anywhere yet. It was an internal talk which I cannot share.)
Fix the ones people blame themselves or each other for. That’s more future proofing than most people do.
Also read from the highscalability blog, there's a lot of experience recorded there. See for example this transcript of an AWS presentation about a step-by-step guide to scaling: http://highscalability.com/blog/2016/1/11/a-beginners-guide-.... Of course it's done with AWS bricks but the ideas are universal
I consult for government departments that often have legally mandated requirements for making something available online. That could be a form submission process, some geospatial information presented on a map, or whatever.
The problem is that when you might have 500-10K user transactions annually, it becomes crazy expensive to write bespoke software, even with the most agile process and the lowest overhead tooling.
Take cloud deployments, for instance. Sure, you can just "click through some wizard" pressing next-next-next-finish and be up and running, but the security team won't allow your infrequently-maintained web server on the Internet without a web application firewall. Setting one of those up is days of tweaking to avoid false positives.
Need to send mail? Azure bans outbound port 25 connections, you have to use Sendgrid, or something like it. Time to read up on yet another unique and special API!
Collecting fees and penalties or making payments to citizens? Woah there... there's a massive API surface you have to learn. Security on top of security is needed. Whitelisted IP addresses. Client certificates. That have to be rotated, manually!
You'll forget some essential maintenance, of course, and then you'll have to set up triggers and alarms so you don't get burned the second time. Which entails mailing lists that change dynamically because the team of contractors has a turnover rate faster than the typical certificate expiry time. Send too many alarms and all recipients will configure an Outlook rule to ignore them. Not enough alarms and you'll miss issues. Just setting this up semi-reliably is an exercise in itself.
Really basic stuff becomes difficult, when you realise that 99% of the alerting and monitoring features in Azure and AWS are designed for systems at their scale. It's all about analysing beautifully smooth curves of graphs aggregating millions of points of data, where deviations are glaringly obvious. These tools are utterly useless when you get one real transaction per day, swamped by a thousand bots. The load balancer health check is 99.99% of the traffic for some of these sites!
Then there's the human element:
Have you tried justifying the time to set alarms for a system where a week-long outage might only affect a dozen customers?
How about the budget to upgrade to a newer operating system for something that is not technically broken -- merely hard to support now?
Or have you tried doing any sort of maintenance on a system that was built by contractors hired for a fixed term, all of whom are now gone?
Meanwhile departments are renamed every couple of years to suit the whims of the latest batch of politicians, so everything has to rebranded. Even tiny little sites use by practically nobody.
I've seen sites up for 10 years, where I estimated that they cost $2,000 per citizen that actually used the site! Madness.
"Brooks insists that there is no one silver bullet -- "there is no single development, in either technology or management technique, which by itself promises even one order of magnitude [tenfold] improvement within a decade in productivity, in reliability, in simplicity."
The argument relies on the distinction between accidental complexity and essential complexity, similar to the way Amdahl's law relies on the distinction between "strictly serial" and "parallelizable"."
Just a few days ago there was this thread about date calculation. One would think how complex can it be? But as one digs deeper they keep discovering layers and layers of special cases.
Take Uber as another example. Push a button get a car was their aim. We see people commenting here that they will be able to build it over a weekend. But the real world of cab hailing has thousands of unwritten, implicit rules that are in people’s head. All of them have to be codified and need to work seamlessly, and on top you add constraints of distributed systems and physics etc.
Or take tax laws, it’s a labyrinth of rules large enough that people have built DSLs.
So, any software that’s solving real world need will be complex. We just have to bite the bullet and deal with the reality.
I think OP heavily overestimates the organisational praxis in other disciplines. Nearly every creative discipline I was ever in was largely purely ad-hoc with very little explicitly stated organisational approaches to the craft. Academia, movie and music production, and creative writing for instance have much less readily-available principles than software.
Software is probably the most thought about creative industry I can think off in modern history.
The decisions you take to build software has to include the expected scale. If you're pushing something to 10 users, you'll take different decisions than if you push it to 20M users. This is done at the planning phase, and senior engineers are aware of this. I'm not going to spend 2 weeks optimizing the shit out of a system that'll be used by 10 people.
In most other creative fields, this can be hand-waved away. No other engineering field has anywhere near the complexity.
Therefore, all important people on a project are stakeholders of some sort, and anyone else is not able to make real improvements.
The project will move in whatever direction satisfies the incentives of the important stakeholders.
I’m a software engineer, so am not qualified to say this, but I highly doubt this is true.
We have good science and steady (albeit slow) progress in the former while it feels like the latter is more or less subject to stagnation. For instance, when I need to setup a DB cluster, why is there no tool that takes my requirements and generates deployment scripts, monitoring, migration tools, etc.?
Sometimes a CEO thinks up a radical crazy idea. While it distributes through middle management it gets twisted. Once it reaches the front line workers nothing really changes. One might consider this inefficient change management but maybe the company protected itself from a stupid CEO decision.
So now we know that one man can build, host and improve/maintain a product himself with the entire earths population as customers.
This is a short window, before thermodynamics takes charge and ruin the prospects, enjoy it while it lasts!
The first one to do Minecraft without hiring anyone or selling it wins. :P
> I have always done my work in the context of a group that is set up to maximize a wide spread of talents and abilities. Just as “science” is “a better scientist than a scientist”, such a group is “a better programmer and systems designer than any individual”. Really learning to program — etc. — is really learning about how to be part of a scientific/engineering/design team. As with a sport like basketball, there are lots of fundamentals that all need to do extremely well, and there are also “special abilities” that not every member will have, but which the whole team will have when it is functioning well as a team.
I fear that very few people really spend the time needed to really learn the fundamentals and very few people learn a scientific mindset that would allow them to cooperate effectively with others without getting their ego in the way.
No mentioning of the underlying infra with all its complexities needed to acheive the goals of flexibility, reliability, speed and cost cutting.
You don't have to be of Netflix size - when you start getting tens of thousands of users, complexity hits you real fast.
Another challenge in software comes from the added cost due to decisions that are tricky to reverse. Worst thing is that they only show at certain scale, meaning that in many cases they were the right decisions in the past (business context).
If you only operate in a single market, you can process prices with a single currency, so you get away with bad modelling - without the currency. To scale to another market, you can just deploy your application twice. If the company succeeds in new markets, you will replicate the solution. Fast forward a few years and you have a migration project to add currency to your software and data to optimize deployment.
Organizational complexity is another domain. Understanding what places to adjust is key. For example: how complex is it to change the tax rate in your software? How do you find out which applications to change? How do you know who needs to perform the change? Do you have to broadcast a question to the whole company and ask them for performing the change until X? How do you know you adjusted all places? Do you have to deploy the change manually or will the change be automatic based on time?
This is only one part of the problem though because as someone stated below even though server systems have networking stacks they are not entirely designed around the concept of seamless computation over multiple nodes so you wind up with abstractions that solve the problem but do so fighting against some occasionally nasty gotchas that make it a lot harder than it needs to be.
This is all to say I’m greatful for those who maintain things like erlang, elixir, kube, go etc. They’re somewhat fighting against the stream so we have an easier time when we do need to scale.
I also -like him- have no answers for "the big picture." I ran a small cog on a big wheel for a long time. I feel the basic quality (as in "few bugs") of our work was good, but I also feel that it took too long, and failed to flex, so it often fell flat with our customers. Our software was developed, using a hardware process, and there are vast differences between the two disciplines.
I am fortunate, in that I don't need to play in anyone else's sandbox. It means that my scope/scale is quite limited, but I am pretty happy with what I am able to do. It's actually fairly ambitious for a one-man "team."
Since striking out on my own, I have experimented with what I term "ultra-agile" techniques; with some success. Hard to codify, though, as they depend on a high degree of experience, as well as the fact that I'm a bit "spectrumish."
The closest to success that I have had, is to design all of my software as an extensible infrastructure. I don't know, exactly, how it will be extended, but I write "hooks" into it. I often end up using these "hooks," myself, in future work.
This right here is exactly why I learned to say nothing in brainstorming sessions and try to dodge then outright if I can.
My experience is these are typically for someone higher up in the pecking order needing their own ideas validated.
My usual work around is if something needs to be done, quietly work on a POC on your own and solicit feedback or even bring in stakeholders.
Software development is hard because "essential complexity" is hard. Whatever people want to do is hard. You can make it easier to write software as much as you want, but you are not making "world" easier. That is main point of Fred Brooks essay.
I see a lot of devs are making statements that now development for them is essential complexity. While no, software and code never will be essential complexity, software only helps us to solve essential things faster.
One of my favorite examples would https://en.wikipedia.org/wiki/Capsicum_(Unix). It's incremental which is good. But people wrote the kernel part and declared mission accomplished like Bush in 2003, ludicrous! One has to do that and then overhaul the userland and either get those patches upstreamed. And these days "userland" isn't a collection of init, shitty scripts and Gtk/Qt, but a bunch of libraries, especially programming language standard libraries.
This would dramatically change the security and ergonomics landscape, because so much global state programmers
"Containers" was always the wrong metaphor because software is about composition, but containers are inert and only their mass and volume composes (very crude). Better to think about plumbing or rail.
Another perhaps more opinionated example is getting everyone to use Nix (or fine, something like it.) Whether with container style virtual global state, or nicer capsicum, we need to make it trivial to install and develop the entire commons. All "builds on my machine but you can use it" just leads to a lack of integration so no one can help the original author smooth over the gaps. It will also allow machines to be demystified, allowing people to toy with all the software on the system, which will help reduce the programmer alienation which allows so much accidental complexity to occur in the first place.
But yeah, almost nobody is doing these things at the scale they deserve, and even the megacorps drown in their own technical debt like sunbelt cities that are just metastasized suburbs. Eveyrone is in the "well I'll eat shit and shut up as long as my competitor is too" mind site. It's disgusting.
Those require different architectural considerations.
E.g. with scale out your chache has to work suddenly in a distributed way.
And as we all know scale-up is expensive, can quickly become very expensive, and more importantly it has upper limits. In most cases scale-out is actually the only option, even if scale-up would work for some while. Many times I have seen architects go the easy route and scale up. They either get promoted or switch companies until their solution either gets too expensive or can't scale any more.
In the meanwhile other systems are interacting with this system and would need to make changes to adjust to a scale-out system.
E.g.: A distributed cache can result in lower write performance, or someone else could have overwritten a certain entry because locking failed (e.g. SELECT FOR UPDATE).
In many cases those systems join the list of legacy systems.
Terrible is a relative term. Terrible compared to what?
Who said/showed (much less proved) that there's a better way and we just don't follow it to achieve it some optimum?
There's also a semantic confusion here. Compared to e.g. the car industry we're infinitely better at "creating software at scale". We can create a billion copies of a software and distribute it our the world, with marginal cost close to zero.
But the author doesn't mean "creating software at scale" like when they say "car production at scale". They mean production of "large software".
Well, let's see the car industry make a car with the scope, flux requirements, shifting environments, etc that large scale software has...
Just allowing the market winners to drive that fashion means we aren’t able to progress the state of the organisational art scientifically.
There were a lot of language zealots at the end of the last century, especially on evangelizing Object-Oriented Programming. Nowadays everybody can easily counter those arguments with 'No Silver Bullet' without further thinking, it's arrogance in disguise of humility. There are still a huge amount of accidental complexities to deal with in most tech stacks. Most businesses would die fast and leave nothing behind anyway, while the progression of the industry would accumulate and benefit the whole industry itself.
Java looks slightly better for creating software at scale than C. C looks slightly better than FORTRAN. FORTRAN looks slightly better than machine code. Say there's a language that looks like Haskell but has tooling and ecosystems as good as Java, I believe it would also slightly better than Java.
> What if software were built in the same way? What if the core parts of our business would be like streets, and all that newfangled stuff is something we could build on top, experiment, tear it down if it does not work?
To me, it kind of already is. The foundation on what we build most software on today; TCP/IP, HTTP, Win32, POSIX APIs, the C-runtime library, threads & processes, etc. Much of that is about 50 years old now.
> Somehow, the code in front of you is just the tip of the iceberg of a lot of mental representation of what is happening
I think this is a key point. When we look at code, the comfort with it is relative to how well we understand that abstraction, and have a good model for what it is doing "under-the-hood". I think a lot of the burn-out and "fatigue" some people feel is where these abstractions are regularly reinvented (I'm looking at you, JavaScript frameworks), and thus you have to spend time learning the details of a new abstraction before you can just read/write a piece of code comfortably.
I can create a startup, get a massive amount of funding, sell the company and make millions of dollars... all without ever seriously caring about performance and scalability.
I would not be hired because I only enjoy executing 'one-thing-well' snippets of code/commands. My minimalism would be a turn-off for most software companies or code shops.
[0] https://bioone.org/journals/acta-palaeontologica-polonica/vo...
I would argue that it is becoming easier for "terrible" software to scale well. I would say that's a much bigger win.
- experienced programmers are promoted out of the tech track - it causes all sort of problems, people keep reinventing the wheel, they are forced to relearn through their own mistakes, transfer of knowledge and skills is hampered
- (probably as a consequence of the first point) people with 3 years of experience are believed to be senior programmers
- worse even - experienced programmers are promoted to non-coding roles aka architects. Over time they increasingly disconnect from the tangible artifact (code) while still hanging onto a false belief that they can function just fine by embracing it through a metaphor (diagrams, etc.) ("Simulacra and simulation"?.. but possibly I'm digressing)
Luckily with software this can be done more easily. We only need to design the new architecture. Design how everything should be executed. Finally we hand over the blueprints and let the computers do all the work.
But it's still going to take some time. A process which is largely invisible. The construction yard is there. But everything is code. You can't really take your stakeholders for a tour around all the work that's happening.
The difference I think is that film-industry has separated the creative parts from the production parts and the creative part is not really complicated at all. Write the screenplay. That can be innovative but it is a single author typically who will do it. No complications because everybody's screenplay has to work together. Because there is only one.
In other words making movies is more like making buildings than making software. Buildings can be huge and very expensive but their design is still quite simple and based on designs the architects have created in the past.
That's sort of what the book says but it's missing the critical part. The whole book is about how to split the solution space into pieces that can be taken "one piece at a time", as it's obviously not the case that just any split will do.
It goes through some real world examples of design and goes on to build a more general semi-formal theory of design, components and independence.
What gives me hope is that there were successful mega projects though not about software. Für example, I read about the Manhattan Project over Project Atlas to the Apollo Project. Those were all mega projects with high innovation and uncertainty like software development. I have no good theory what makes it successful though.
What distinguishes software from other techne is a lack of physics. There is no 'solid ground of reality'. All other forms of making involve discovering and then applying the governing laws.
Hardware architectures, operating system, and programming languages, to an extent, do furnish a phenomenal context and where these characteristics are stable and well established (e.g. Memory Models), science appears [1].
But clearly, this ground of reality is indeed 'soft' and the practitioners usually re-invent not just the wheel, but the ground that it rolls on as well.
The dilemma -- which nearly means facing a choice of a multiplicity of lemmas -- that confronts the designers of hardware, OS, and languages is precisely that tension between generality (and corresponding weak "world" semantics) and specificity (with robust but constrained semantics).
[1]: https://ocw.mit.edu/courses/electrical-engineering-and-compu...
I thought the test-driven development idea was impractical and pedantic when I first heard about it. I gave it a shot, though, and the thing it allowed me to do was to prevent massive systems from becoming impossible to modify. When a system gets so big that it is impossible to fit it in one's mind, most developers get scared to do new releases because there's probably some critical part of the system they forgot. They forgot all the requirements and QA test cases for at least one obscure part of the system. Even if all the QA test cases are documented, the process to release becomes increasingly difficult and time consuming.
With test-driven development, you can just run all the tests, and if they pass, say ship it. The key is not to get too many tests, especially integration tests which take a long time to run and like to break when designers make UI changes. Usually, I start with the happy path integration test and then write tests as I'm developing for things that don't work right. About 40% of the stuff works the first time, and I never have to write a test for it. 45%, I write the test and fix it, and it passes, and I can forget about it. The other 15% is usually some tricky algorithm with many corner cases, and that's where most of the testing goes. I typically write one happy path integration test and then fill in the lower level tests as needed. When the happy path works, most of the system is working. That said, spending time on integration tests is usually a massive waste of time. It either works the first time or something lower down broke, and one should write a test for the lower level part that runs in a shorter amount of time than the integration test.
I was able to port a large enterprise app from an Oracle backend to Postgres because there were tests for everything. The port just amounted to getting all the tests to run on the new database and the necessary data migration. This migration was by no means a trivial feat, but it was at least possible, and it ended up working and saving the company millions in license fees.
The point being. A system with millions of lines of code is approachable if it has good tests. A new developer can work on it, and if all the tests pass, that developer didn't break anything. I can go back to the code I wrote years ago and still use it if I run the tests, and it works. I can then see how it's supposed to work, add features, and so forth. Without tests, this is very difficult because as systems become bigger, they break more and more if they don't have tests.
If you take processes from people who build machines and apply them to software, it will fail. Software means inventing machines.
Maybe if software "engineers" were treated more like actual engineers then we would start seeing better results.
First, consider a car. Imagine if somebody told you that a random company would be changing the software that runs your car, every day, as you drive it. Freaked out? You should be. That's what a modern tech company is. They just don't have the same risks, so they don't care so much if they fuck up, so the work isn't organized very well.
Next consider cities. Do you want to build something? Great. I'll ignore all the costs (there are many).
First you're gonna need a design by an accredited expert. That design needs a dozen permits approved by the government before you can even touch a trowel or hammer. Then a team of experts guess to town, changing grade, pouring foundation, waiting to cure, installing basic plumbing, and then getting an initial inspection. Then comes framing, inspection, plumbing, inspection, electrical, inspection, HVAC, insulation, drywall, fixtures, trim, walkways, driveways, flooring, landscaping, grading, and inspection.
So far we have used specialist teams to build sections of the building and only continue when a strict inspection by authorities says we can continue. No customer has used the building yet. Also consider that this building will not scale. If you reach capacity, go make another building.
Building software at scale is a combination of monkeying around with a car's internals while someone is driving it, and building a building. Only the risks to human life are lower, and we do not have teams of very specific contractors strictly following government approved codes and zoning laws to build one thing that meets one set of criteria.
The people doing the job(s) in tech aren't good enough at it to do the things we're asking of them. And they aren't just building one piece and walking away, they are constantly fiddling. And very often, they have no master plan approved by inspectors according to well known and inviolable codes.
On top of that, they have never sat down and figured out how to do all of this really efficiently amd reliably. Scrum/Kannan are general processes; they do not tell you specifically how to build a website in an extremely efficient and error-free way. But we've done that a million times by now, poorly. It's because we haven't yet codified it as an engineering discipline and stripped out the fat. And we haven't done that because there's no requirement to, because nobody's life is on the line, the way it is with cars and buildings.
And it's hard. It's hard because we still hand-make components rather than buy them off the shelf. Every company I've worked at has re-created the same god damn thing 1000 times, by hand, because for some bizarre reason they thought it was a better idea to forge steel than to connect plumbing. Imagine if your plumber hand-crafted her own custom fittings for each job.
Really, we do a marvelous job today considering how completely undisciplined, unregulated, haphazard, and dangerous what we do is. There are many approaches we could take to simplify the actual process of it all, and make it efficient. But the complexity and difficulty will be there for a very long time. We can also move to pre-fab, but you'd have to convince people that writing new software just to build a tech product is a bad idea. Good luck with that.
I propose that scaling (as defined here) is a general problem, not just a software problem. I realize this is a hard case to make. Technical debt, refactoring hells and such don't plague film sets or factories like they do code shops. But.... Consider this:
- Consider factories. A factory is just a huge pile of physical capital organised to efficiently do a specific, defined task. Everything is optimised to reduce marginal costs. Capital efficiency is strictly enforced by capital money markets. Marginal cost efficiency is strictly enforced by real markets.
... Either of these two constraints (marginal costs too high or not enough capital) usually come into play before engineers have had a chance to redesign, refactor, repurposed and abstract a factory to the point where these are a problem.
- Everything that is in a movie is in a shot. Movies have a finite number of shots and a director can direct each one. A film may have a whole team replacing Toby Philpott with Jabba the Hutt, an army of costume designers, set designers, actor psychologists, etc. But, everything that gets into the movie gets there in a shot, and the director can work shot-by-shot and wield god-like control of lots of labour that way.
Software is unique. Software is free from most physical limitations. There are no material costs. There are no capital costs. The only economic resource that a software enterprise wields are the engineers/engineering itself.
When a Ford Company scales, it raise money. It uses that money to to build factories, buy materials, etc. These are all scarce/finite resources needed to start/continue making additional cars.
Google, MSFT, FB, post-AWS amazon... When they scale up, they just scale up. They hire more engineers. They produce more software. The only "resource" being scaled up is the people making the software.... Something has to be the limiting factor.
In any case, the "scaling creative work" problem does exist in factories too. The difficulty 50s era auto manufacturers faced competing with Toyota are sort of evidence. They struggled with "technical debt" in the form of car models, factory design, company culture and such that couldn't adapt flexibly.
A lot of tesla's wins (besides marketing, fundraising and software) have come from their recent blank state start. Auto manufacturing today is highly caught up in the "Toyota Way" of doing things. It's been that way for decades and parts of it are explicit assumptions of international trade deals. It's very compartmentalized. Flexible within compartments. Rigid without compartments, whether they are outside the department or outside the company. It's a lot like the city metaphor this article mentions.
The problem is... sometimes you need to design the factory and the car simultaneously. At times that has screwed Tesla. The quality control benefits of the Toyota Way are not to be taken lightly. At other times though, it works. The factory designer has been banned from designing the car and the inverse for 40 years. I think their version of technical debt is the reason.
