"Complexity can neither be created nor destroyed, only moved somewhere else."
The drive to simplify is virtuous, but often people are blind to the complexity that it adds.
Okay, so your microservices are each very simple, but that made the interactions and resource provisioning very complex. What was the net gain?
The correct solution depends on the circumstances. There are excellent uses of microservices. There are excellent uses of monoliths. There are excellent uses of monorepos. There are excellent uses of ... (wait never mind monorepos are just better).
Understand what is ESSENTIAL complexity and what is INCIDENTAL complexity. Simplify your system to remove as much incidental complexity as possible, until you are left with the essential and be satisfied with that.
The quote would probably be more accurate as:
> "ESSENTIAL Complexity can neither be created nor destroyed, only moved somewhere else."
That is the more precise adaptation of Kelsers Law.
Obviously you can always add also superfluous complexity :)
Every time I've had that argument at work, ultimately there's no agreement of what is essential. In one of the worst cases, the big fan of the concept was a CEO considering themselves very technical, who decided that everything they didn't understand was obviously valueless and incidental, while everything they personally cared about was really essential complexity, and therefore impossible to simplify... even though the subsystem in question never had any practical applications until the company failed.
So ultimately, either the focus on avoiding incidental complexity is basically a platitude, or just a nice way to try to bully people into your favorite architecture. A loss either way.
I have to say that this short quote is not the whole story; for example it's ridiculously common for artificial complexity to be introduced into a system, like using microservices on a system that gets 1k users a day.
In which case, it is sometimes possible to remove complexity, because you are removing the artificial complexity that was added.
The problem is that the shops that move from monoliths to distributed systems are under the impression that all of their problems are now magically over "because microservices".
This sort of specious reasoning just shows how pervasive is the fundamental misunderstanding of the whole point of microservices. Microservices solve organizational problems, and their advantages in scaling and reliability only show up as either nice-to-haves or as distant seconds.
Microservices can an do make sense even if you have 10 users making a hand full of requests, if those services are owned and maintained by distinct groups.
The net gain was composability of microservices, distribution of computing resources, and the ability to marshall off implementation details. Just because those requirements were routinely ignored in the era of monoliths doesn't mean the complexity wasn't essential or didn't exist.
What microservices give you is a hard boundary that you cannot cross (though you can weaken it, you cannot eliminate it) between modules. This means the internal state of a module now has to be explicitly and more deliberately exposed rather than merely bypassed by lack of access control, or someone swapping out public for private, or capitalizing a name in Go. If there's any real benefit of microservices, this is it. The hard(er) boundary between modules. But it's not novel, we've had that concept since the days of COBOL. And hardware engineers have had the concept even longer.
The challenge in monoliths is that the boundary is so easily breached, and it will be over time because of expedient choices rather than deliberate, thoughtful choices.
But isn't that introducing incidental complexity? Not sure if you actually disagree.
The main misconception about microservices is that people miss why they exist and what problems they solve. If you don't understand the problem, you don't understand the solution. The main problems microservices solve are not technical, but organizational and operational. Sure, there are plenty of advantages in scaling and reliability, but where microservices are worth their weight in gold is the way they impose hard boundaries on all responsibilities, from managerial down to operational, and force all dependencies to be loosely coupled and go through specific interfaces with concrete SLAs with clearly defined ownerships.
Projects where a single group owns the whole thing will benefit from running everything in one single service. Once you feel the need to assign ownership of specific responsibilities or features or data to dedicated teams, you quickly are better off if you specify the interface, and each team owns everything begind each interface.
If team A needs a new endpoint from team B, what would a typical dialogue look like under microservices and a modular monolith, respectively?
"For something to get clean, something else must get dirty"
and the corollary:
"You can get something dirty without getting anything clean"
I have absolutely seen complex code that was created (often for perceived “best practices” like DRY) which could be removed by simplifying the code.
I think a good solution to any problem needs to match it in complexity. I regularly use this comparison as a benchmark for solutions.
Of course, you can also see it this way: The complexity you remove from the code by making it cleaner is added to your team communication because you now have to defend your decision. (Only half joking.)
Succintly put!
I believe we do not have the right tools yet.
Not true. I blow.up a car. Complexity is destroyed. I rebuild the car from blown up parts. Complexity is created.
There's no underlying generic philosophical point about microservices and monoliths. What we can say is that microservices are not necessarily less complex than monoliths, but this relationship has no bearing on the nature of complexity itself.
Wouldn't the blown-up car be more complex than the non-blown-up car?
I don’t know how many of you are musicians, but many people are surprised to find that the tone quality of a guitar has way more to do with the person playing it than the way it was built.
Framework solves a general problem, but we try to solve a specific problem, thus it will always be a mismatch, frameworks usually solves that with a combination of magic and opaque abstraction layers.
Frameworks works best for short running projects, long running projects tend to outgrow the framework.
Prefer libraries over frameworks.
Personally if I was "stepping up" from PHP and wasn't going to use something powerful (or didn't need all that power) I'd be stepping up into javascript and nextjs and stuff like that.
Instead, factor services along natural fault lines. These are areas in the solution that scale differently from other parts and can tolerate communicating over http or message queue.
It is fine to have lots of people work on a single service. We compose things using 3rd party libraries all the time. Just treat internal code a bit more like 3rd party libraries.
More like 3 people requiring 13 microservices
That's not to say there's no relationship between team size and the applicability of a service-oriented architecture. Microservices are a way of drawing hard boundaries around blocks of logic. These boundaries come with cognitive and operational costs, so they represent significant overhead, but they are a tool for abstracting both logic and physical operations to the maximum degree possible (100% is never possible for a single application). In order to get a net benefit, you have to have enough engineers that they can be experts in a subset of services, and the interfaces to their peer services have to be reliable and stable enough that they can be productive without knowing anything about their internals. So while I agree with you there is no universal floor of microservices that makes based purely on team size, there definitely is a ceiling.
Is there though? Operating systems have 10K - 20K devs based on a casual internet search. Of course there are many services / processes but they are not communicating using an http API or message queue. Similarly, Postgres has 350 people. I don't think team size is a factor at all.
Is this something that actually happened? Not heard from some third party - actually has someone experienced this as a decision in a technical team? It seems... unlikely.
Multi-team ownership of a single deployable has downsides to tradeoff against the complexity of more service and harder boundaries.
(Not saying it's the right way to do things, just the natural way to optimise if you need to keep 20 teams busy working on one system. I think the incorrect assumption is that all 20 teams need to be busy doing the same thing.)
Here's why I'd choose microservices for a large project:
1. People don't produce uniform code quality. With microservices, the damage is often contained.
2. Every monolith is riddled with exceptional cases in a few years. Only a few people know about corner cases after a few years, and the company becomes dependent on those developers.
3. It's easier for junior developers to start contributing. With a monolith you'd need to be rigid with code reviews, whereas you could be a little lax with microservices. Again, ties into (1) above. This also allows a company to hire faster.
4. Different modules have different performance and non-functional requirements. For example, consider reading a large file. You don't want such an expensive process to compete for resources with say a product search flow. Even with a monolith, you wouldn't do this - you'd make an exception. In a few years, the monolith is full of special cases which only a few people know about. When those employees leave, the project sometimes stalls and code quality drops. Related to (2).
5. Microservices have become a lot easier thanks to k8s and docker. If you think about it, microservices were becoming popular even before k8s became mainstream. If it was viable then, it's a lot easier today.
6. It helps with organizing teams and assigning responsibility.
7. You don't need super small microservices. A microservice could very well handle all of a module - say all of payments (payment processing, refunds, coupon codes etc), or all of authentication (oauth, mfa etc).
8. Broken Windows Theory more often applies to monoliths, and much less to microservices. Delivery pressure is unavoidable in product development at various points. Which means that you'll often make compromises. Once you start making these compromises, people will keep making them more often.
9. It allows you the agility to choose a more efficient tech/process when available. Monoliths are rigid in tech choices, and don't easily allow you to adopt a different programming language or a framework. With Microservices, you could choose the stack that best solves the problem at hand. In addition, this allows a company to scale up the team faster.
Add:
10. It's difficult to fix schemas, contracts and data structures once they're in production. Refactoring is easier with microservices, given that the implications are local compared to monoliths.
1 & 8: This is something I've experienced first-hand quite a bit. When code quality starts slipping in service, it accelerates and inevitably leads to more hacks built on hacks. The containment afforded by service separation helps keep these hacks quarantined and minimizes the pollution and spread.
6. There's no ambiguity or question of who owns what when services/packages are separated and have distinct owners. Transferring ownership means transferring ownership of the service. In large companies with some teams working on secret projects, it's much easier to have viable workflows with service separation than with a monolith. Permissions management and so many other things are easier.
7: I think this is an excellent point. Another way of looking at this is as having multiple focused monoliths.
I'd also add that there's a benefit in blast radius minimization. Updating a library or merging in a risky change doesn't risk breaking literally everything all at once. One service can update something and you know for a fact that it can't break anything other than that service and its dependents.
Point 7 is the most important of the technical considerations: just make your services big enough to make sense as a separate unit and small enough to not be another unchangeable monolith. Yes, it’s not sane to have 300 Lambdas that add one number to another talking to each other over network, so just don’t do it.
Microservices the “Netflix way”, with the huge graph of services, gives a bad name to the idea of factoring out modules into independently deployable parts, which always made sense. Kubernetes just helps with deployment, but how coarse or fine you factor is on you.
The simplified view I hold is this: if your main problem isn't a lack of engineers, but how to split work between hundreds, or even thousands of engineers, then microservices make a lot of sense; that's true at FAANG and some other companies. If your main problem is how to get the maximum out of a limited number of engineers you can afford to hire, then microservices is not the solution. The latter is by far the most common case.
We weren't doing Sun RPC, PDO, CORBA and DCOM everywhere for fun, yet with some rebranding, SOA should be the solution for everything.
Monoliths vs microservices is really a spectrum.
Microservices trade design complexity with operational complexity, to address the shortcomings & patterns of the modern IT sector, at both worker and management levels.
This was an excellent choice for consulting (companies like ThoughtWorks, who were pushing microservices) as it optimized their workflows and bottom line: the answer to every client's question of "what's the architecture?" was "Microservices". The companies pimping m.s. were internally optimized for this architecture - running a consulting shop where the time between "here is our project" to "here is our proposal" is approaching zero is profitable.
And the best part -- just like all those "GPT-processes" now being pushed by companies built around an api -- is that by the time clients encounter operational complexity of running micro-services, the consultants have long left the building.
Yup!
And it annoys me so much how architects/consultants get away with just proposing 'microservices' without being held responsible for all the increased operational complexity. It's left as something for devops to figure out later.
Never seen it work, don’t think I ever will. The only way microservices can ever work is if they’re extracted over time as the monolith gets fleshed out, likely over the course of years. And even they’re probably a bad idea.
The microservice syndrome begin once you start splitting services simply because you arbitrarily declare them too large.
Determining this at all takes skill. Determining it that far ahead of time requires so much skill and foresight that suggesting that people try is bad advice.
Otherwise, I generally agree... I'll usually take a monolith approach and break things off in ways that make sense. Usually starting with long running processes that can simply be workers off of queues. Sometimes potential bottlenecks that have higher compute overhead, such as passphrase hashing and comparison which is relatively easy to DDoS, but if broken off only effects new logins and password changes.
In order to DoS your typical site through passphrase hashing you would need to be:
- have a ton of valid usernames/emails of accounts that need to be checked (because a typical password check will rate limit by account) - send in a massive torrent of traffic from a ton of IP addresses (because a typical password check will be rate limited by IP, even more than typical IP based rate limiting)
While this is not impossible if you had those resources it still might be easier to just DoS the site though standard pages/ endpoints by sheer traffic.
So I generally try to design with a separated authentication from the beginning. I'll start with a "dev" auth service that will just have a form you can fill out whatever you want for the "user" and permissions, then sign a token to get into your local/dev environment application. From there it's really easy to create a more robust system or wrappers for external systems (Auth0, Okta, Azure AD, etc).
In general, however, MicroServices is about completely separating a given context domain from other services in a larger system. Account Management is separate from Assets, etc. In practice this means you will have a much more complex orchestration, deployment and communications system in place, often with some sort of Distributed RPC, Queue or other layers on top. You also have to be much more concerned with communications between teams and service versioning and availability.
This complexity isn't less complex, it just shifts the complexity, which can help with larger organizations, but for smaller teams it can really bottleneck everything. This is why the general concensus is to start with a more monolithic codebase designed to still scale horizontally, then break off systems as the need arizes.
The exception being if you are in an organization that has already paid the overhead/debt of setting up for microservice orchestrations.
It's only natural that a job that used to take 2 now takes 10 people because of those facts.
And I agree. We've traded safety and a few other things that could have been ironed out in exchange of massive gridlocks, headaches and colossal facepalms. Performance is only on par if you consider the hardware improvements, it's crazy.
I am already seeing this newly discovered love of simple, fast monoliths and the developer ergonomics they offer. What is old is new again - we are at the beginning of this cycle. The older generation learned that "complexity kills", the new generation is beginning to get it as well.
The systems can be worse because of what other commenters mention, the financial incentives, but also the companies profiting from worse systems must ensure that expectations are gradually lowered amongst people who write software and that these peoples' "skills" are progressively dumbed down. That is how they ensure that status quo is maintained and that people will never "go back".
Meanwhile, hardware and networking improvements have been amazing.
DHH has a couple of posts on the topic recently: https://world.hey.com/dhh/how-to-recover-from-microservices-...
1. Don't pick an architecture because it's all the rage right now.
2. Don't pick an architecture that mimics your org's structure. Aka don't fall prey to Conway's law: https://en.wikipedia.org/wiki/Conway%27s_law.
3. Don't pick an architecture that your team can't operationalize--e.g. due to lack of skills or due to business constraints.
You can't not fall prey to Conway's law. You can only choose how you organize your people and their interfaces.
This is natural and even beneficial: it reduces communication and coordination costs, which is the costliest and hardest part of any organization or system. Perhaps that sometimes lands you in a suboptimal local minimum, but escaping that is usually quite costly in the short term. Think about how often top-down reorgs actually generate value.
The only counterexample to Conway that I have seen working is “away teams”:
https://pedrodelgallego.github.io/blog/amazon/operating-mode...
* "I always urge builders to consider the evolution of their systems over time and make sure the foundation is such that you can change and expand them with the minimum number of dependencies."
* "There are few one-way doors. Evaluating your systems regularly is as important, if not more so, than building them in the first place."
This is useful advice to people who can make high level org decisions. But they don’t necessarily know what’s going on at a software level.
The people who do know what’s going on often have a very hard time getting buy in for refactoring. Many of them (rightfully) conclude it’s better to just make management happy churning out features.
> If you hire the best engineers....
Guess what, there is no broad consensus on what "best engineer" means. I bet your org is rejecting really good engineers right now because they don't know what Kubernetes is. Same goes for literally any other technology that has been part of a hype cycle (Java in 2001, Ruby on Rails in 2011, ML in 2011; no the precise years don't matter).
> ...trust them to make the best decisions.
A lot of work encapsulated there in less than ten words. If you hire a bunch of people and tell them "you are the best", you think they are going to sit around and run the Raft protocol for consensus on deciding how to architect the system? No, each of them is going to reinvent Kubernetes, and likely not in an amazing way.
Microservices are often best deployed when there is a mixture of cultural and engineering factors that makes it easy to split up a system into different parts that have interfaces with each other. It has little to do with the superiority of the technical architecture.
----------------------------------------
[1] Looks like the article was written by the CTO of Amazon, which...surprises me a bit. Then again, from all accounts, Amazon's not exactly known as the best place to work; so maybe I'm right? In any case, anything written by Amazon is not directly applicable to the vast majority of small-to-medium companies.
In companies I'm familiar with projects are run by technical leads and product leads.
Managers don't interact with anyone doing actual work, which is probably for the best.
Also, they never claimed technical superiority of microservices, quote:
> For example, a startup with five engineers may choose a monolithic architecture because it is easier to deploy and doesn’t require their small team to learn multiple programming languages.
That’s weird. Any time I’ve seen anyone let their engineers engineer, those engineers have immediately started following the hype.
No, but they are arguing against the strawman of the perceived technical inferiority of monoliths. Look at the title of the article.
I am simply calling out that strawman as such.
If they wanted to convey the message of "It all depends, use the best architecture for the job", they should reflect that in the title.
I've seen terrible spaghetti code apps where literally nothing can be refactored because of the model / view, God object dependency stuff all over the place.
If the need arises, you could move into SOA, and break out a couple of your larger domain modules. Continue following these same rules though.
If the need arises still, you could move into micro services, and break out more / all of your domain modules. Truly understand first whether you actually need this first.
But the "let's do micro services this monolith is old junk" trope, abandonwaring the codebase, building out a bunch of services without strong, fundamental domain knowledge, and then complaining when shit is expensive and fragile and broken- it's getting tiring.
It would be awesome if you could set up a parallel “monolith-stories.com” site with the exact same content, except for inverting the sentiment scale :)
Seriously!!
Like is it two pizzas per microservice or 10 microservices per one pizza.
Nearly spat out my coffee. Thank you for making my morning.
Increasing number of pizzas you get bigger teams.
Increasing number of microservices you assign more of them to a single team.
It's really only Amazon themselves that can afford to do so.
So Netflix doesn't really count as an example of how AWS is a practical option for video hosting.
Just pick the right architecture for the given problem. Sometimes it's monolith, sometimes it is not. The end.
Obviously doesn't scale but aluring anyway
[1] Too lazy to keep looking right now
[2] https://en.wikipedia.org/wiki/Conway%27s_law
[3] https://www.reddit.com/r/ProgrammerHumor/comments/6jw33z/int...
As money becomes more expensive and as we inch further towards a massive economic crisis, companies that have allowed their R&D budget to bloat out of proportion with needlessly-distributed architectures are NGMI.
What even is a microservice?
There are other distinctions, borders and splits that are more important to consider.
Here's a koan that highlights a few of these considerations:
> If you run a kubernetes cluster on a single physical server, is it a monolith or is it a microservice architecture?
It’s not “monolith vs distributed”.
It’s “good monolith vs bad monolith or big ball of mud vs domain-driven design”
It depends on what your primary domain is, level of complexity, number and makeup of enterprise integrations, and more.
Some monoliths are very bad.
Some distributed systems are very bad.
My rundown is:
- is it a simple crud system? —-- monolith
Otherwise: - model it, identify bounded contexts, proceed accordingly
The last hyper growth startup I worked at grew 10x in scale every 2-3 quarters for nearly 3 years at meaningfully large scale (millions of monthly transacting users). In that time, the number of different business-lines/categories and amount of functional flows and their intersecting/overlapping complexity also grew multiple folds.
So, we were adding whole new things and throwing away old things and basically refactoring everything every 18-months. Without knowing consciously, the superpower we had was our ability to refactor large live systems so well. In hindsight it became clear to me that our ability to do this hinged on a few different things:
1. A critical mass of engineers at both senior and junior levels understood the whole systems and flows end to end. A lot of engineers stayed with their own team developing strong functional-domain understanding. Similarly a good number of senior engineers rotated across teams.
2. The devops culture was extreme – every team (of 10-12 engineers) managed all aspects (dev-qa-ops-oncall etc) of building and operating their systems. This meant even very junior engineers were expected to know both functional and non-functional characteristics of their systems. Senior engineers (5-10 yr experience) were expected to evaluate new tech stacks and make choices for their team and live with the consequences.
3. Design proposals were openly shared and sought critical feedback. Technical peer reviews were rigorous. Engineers were genuinely curious to learn things, ask and understand things, challenge/debate things etc. Strong emphasis on first-principles thinking/reasoning and focusing on actual end-to-end problem-solving without being territorial or having dogmas was strongly encouraged and the opposite was strongly discouraged.
4. Doing live-migrations – we mastered the art of doing safe live migrations of services whose API schema or implementation was changing and of datastore whose schema or underlying tech was changing. We had a lot of different database tech migrations – from monolith SQL dbs, to NOSQL clusters to distributed SQL dbs and their equivalent in-memory dbs and caches.
Surprisingly, the things we didn't do so well but didn't really hurt our ability to refactor safely were:
1. Documentation – we had informal whiteboard diagrams photographed and stuck in wiki pages. We didn't have reams and reams of documentation.
2. Tests – we didn't have a formal and rigorous test coverage. We had end-to-end tests for load-testing and we had a small manual QA for doing end-to-end integration testing for critical flows. These came about a bit later – but trying to scale them effectively proved very challenging. But these were not seen as hurdles for doing refactors.
3. Formal architecture councils and formal approval processes – we didn't have these. Instead we had strong people to people connect and strong team level accountability – culturally people owned up their mistakes and do everything they could to fix things and do better next time. Humility was high.
Later, I worked at a large mature company with very large scale – and everything was exactly flipped on all the points above and any major refactors were a serious pain and migrations took forever and actually never completed. The contrast was very eye-opening and I realized in hindsight the above contributing factors.
He’s been ranting against cloud too.
[1] https://world.hey.com/dhh/even-amazon-can-t-make-sense-of-se...
