Most startups - most large companies - would be far better served with a real PAAS, rather than container orchestration. My encounters with container orchestrators is that ops teams spent inordinate amounts of time trying to bend them into a PAAS, rather than just starting with one. This is why I don't understand why this article lumps, e.g. Cloud Foundry in with K8S - they solve entirely different problems. My advice to almost every startup I speak to is "Just use Heroku; solve your business problems first".
The article also mentions it enables "new set of distributed primitives and runtime for creating distributed systems that spread across multiple processes and nodes". I'll throw out my other assertion, which I always though was axiomatic - you want your system to be the least distributed you can make it at all times. Distributed systems are harder to reason about, harder to write, and harder to maintain. They fail in strange ways, and are so hard to get right, I'd bet I can find a hidden problem in yours within an hour of starting code review. Most teams running a non-trivial distributed system are coasting on luck rather than skill. This is not a reflection on them - just an inherent problem with building distributed logic.
Computers are fast, and you are not Google. I've helped run multiple thousand TPS using Cloudfoundry, driving one of Europe's biggest retailers using just a few services. I'm now helping a startup unpick it's 18 "service" containerised system back to something that can actually be maintained.
TLDR; containers as production app deployment artefacts have, in the medium and long term, caused more problems than they've solved for almost every case I've seen.
Kubernetes helps with one thing: taking your container and running it on a fleet of machines.
Building 18 services is an architectural choice made by the team. It has nothing to do with containerization or Kubernetes. For a single team, a monolith just works most of the time. You may consider multiple services if you have multiple [large] teams, think Search vs. Maps. Even then, consider the trade-offs carefully.
The build server creates one zip file that is stored as an artifact that gets decompressed and released in each environment - in a separate folder.
An EAR packaged with everything needed by the application.
Each service, or micro-service as it is fashionable now, got their own EAR.
Deployment of UNIX based OS, JEE application server, Oracle and respective EAR packages, done.
I think that for reproducible development environments is a much simpler solution.
Here's the path that leads to K8s too early.
1. We think we need microservices
2. Look how much it will cost of we run ALL OF THESE microservices on Heroku
3. We should run it ourselves, let's use K8s
One of the big "Elixir" perks is that it bypasses this conversation and lets you run a collection of small applications under a single monolith within the same runtime...efficiently. So you can built smaller services...like a monolith...with separate dependency trees...without needing to run a cluster of multiple nodes...and still just deploy to Heroku (or Gigalixir).
Removes a lot of over-architectural hand-wringing so you can focus on getting your business problem out the door but will still allow you to separate things early enough that you don't have to worry about long term code-entanglement. And when you "need" to scale, clustering is already built in without needing to create API frontends for each application.
It solves a combination of so many short term and long term issues at the same time.
Elixir and Erlang are basically a DSL for building distributed systems. It doesn't remove all of the complications of that task, but gives you excellent, battle tested, and non-proprietary tools to solve them.
This is also true of Erlang, for those not aware that Elixir runs on the Erlang Virtual Machine (BEAM).
You do get a lot of cool things with clustered nodes though (Node monitors are terrific) and tools like Observer and Wobserver have facilities for taking advantage of your network topology to give you more information.
They are basically an OS, with containarized applications.
Thanks to them I stopped caring about the underlying OS.
One day I spun up a k8s cluster on GKE and just started putting all projects there. This cluster enabled huge cost savings (running a fleet of 3 VM's instead of ~50), allowed cheap per-feature dev/staging environments, forced developers to consider horizontal scaling BEFORE we needed to scale (read: when we missed our only shot), and overall reduced ops workload tenfold. It wasn't without a few challenges of its own, but I would never go back.
For devs stuck in a silo it's a little like putting margarine on butter. For DevOps looking at hundreds of little silos it's the foundation of operational sanity.
Parts of the system were distributed not for capacity, but for HA reasons. So where before we had two instances of beanstalkd with their own storage and clients had logic to talk to both, we now have a single instance of beanstalkd backed by distributed storage and a Kubernetes service that points to it.
And I think we get more benefit deploying dependencies than we do our own apps. If one of them is low volume and needs mysql, just `helm install mariadb`. No complicated HA setup, no worries about backups, we already know how to backup volumes.
I agree completely and your comment gives me the perfect opportunity to praise how much I love the flexibility of Hashicorp's Consul+Nomad.
Nomad let's you run almost anything - Docker containers, executables (the raw_exec driver), jar files, etc.
https://www.nomadproject.io/docs/drivers/index.html
Dead simple to setup - one self contained < 20Mb executable that can be used in either client, server, or dev mode (client + server), configuration is basically automatic as either a single server or cluster of you are using Consul.
The stock UI is weak but The third party HashiUI is great.
It's also a thousand times easier to maintain. (Thanks CoreOS) Basically if you want a managed simple no maintance, no cost bare-metal K8S installation you basically just use tectonic/kubeadm and you get something which is self-containing, or close to self-containing. and the only things you need to get it done is actually way easier than reading through cf docs (I'm pretty sure bare-metal isn't even supported that easily).
running some services on top of it is than pretty simple, especially if you want to use a single ip, insteand of roundobin dns (https://github.com/kubernetes/contrib/tree/master/keepalived...)
and if you have k8s running, adding some PaaS layer on top (openshift) can be pretty simple.
BOSH with the RackHD CPI does this. It's the same basic operator experience across every platform with a CPI.
Disclosure: I work for Pivotal, we work on this stuff.
https://cloud.google.com/kubernetes-engine/
https://azure.microsoft.com/en-us/services/container-service...
https://www.ibm.com/cloud/container-service
Or have someone knowledgeable build the service for you.
You mention 18 microsevices, I think that small teams are better off with a monolith.
I would see Kubernetes as a new machine level. We're moving from bare metal, to VMs, to container schedulers.
Heroku was one of the first companies that ran a container scheduler internally. So I think we agree that is the future.
But a small team probably doesn't need to work at that abstraction level.
At GitLab we think most teams will want to work at a higher abstraction layer. Just push your code and have it deployed on Kubernetes. Without having to write a dockerfile or helm chart yourself.
A lot of people forget that you can just put your application up on 1 server and serve hundreds of thousands or millions of requests a month without breaking a sweat.
For that type of use case (1 box deploys), Docker is still amazingly useful so I would 100% containerize your apps for that purpose, but I agree, Kubernetes and container orchestration in general is overkill for so many projects.
That said, a good number of projects don't require anything special about the environment other than a runtime for the app's language, where the remaining dependencies can be explicitly included in the build. For those, I agree, jumping on docker/k8s right away is overkill.
An additional benefit of working with something like Heroku initially, is that it will help guide your devs to sticking with more tried and trusted stacks rather than everyone pulling in their own pet project into the business's critical path.
As a nitpick:
> This is why I don't understand why this article lumps, e.g. Cloud Foundry in with K8S - they solve entirely different problems.
In fairness, the reference was to Cloud Foundry Diego, which is the most analogical component to Kubernetes. And they are of comparable vintage. Diego never found any independent traction outside of CFAR.
> I've helped run multiple thousand TPS using Cloudfoundry, driving one of Europe's biggest retailers using just a few services.
We have customers doing millions of payments per hour, billions of events per day. Running tens of thousands of apps, thousands of services, with thousands of developers, deploying thousands of times per week.
CFAR doesn't get much press out of enterprise-land, but it works really well.
Disclosure: I work for Pivotal. We have commercial distributions of both Cloud Foundry (PAS) and Kubernetes (PKS).
So there's all these layers of stuff that sit between the user and the data just to make the act of asking WHERE DATATHING="STUFF" convenient.
There's a lot of work (and money) out there to fix systems implemented on the hype train.
The batteries included nature of swarm is a huge help as well - with k8s, you have to muck around overlay network, ingress, etc.
However, I think the writing is clear on the wall - k8s has won. Probably even to Docker Inc, given the kubernetes integration they are building into swarm now.
I think Docker Swarm can exist as an opinionated distro of k8s. I wouldnt mind paying it money for that.
We don't use Kubernetes at my shop, we've begun to use OpenShift though which layers PaaS tooling on top of it and the developers on my team love it. They create a deployment, point it at the git repository containing their code, set their configuration and the app is live - the underlying primitives are available if we need them still, but that's for me to worry about as the DevOps guy and not the developers.
Helm (https://helm.sh/) allows you to define an app as a collection of K8S components then to manage (=deploy, update, ...) your app as a standalone component
If you are looking for “I just wanna run my app” I found CloudFoundry to be dope among all the other PAAS solutions out there.
I migrated a decent sized solution still in dev from SF to .netcore and SF - 10/10 would do it again. Not to mention that you also end up saving 50% $$$ on vm costs with linux vms (not considering SF on Linux)
The next shift is what I've called "stackless" - why do you even care what platform it runs on?
All you want to be able to do is have your application run somewhere.
Kubernetes goes some way towards that, but there's another abstraction layer needed.
Similar to how Docker was an abstraction further to Kubernetes and away from Vagrant.
This is something I wrote about this not long ago[1].
1. https://wade.be/development/sysadmin/2016/11/17/stackless.ht...
Serverless isn't a good name - but it doesn't stand for "don't care about the hardware". Devs are already not caring about hardware anymore since VMs.
What serverless removes is the abstraction level of a server/vm/container.
A simple example is scaling your stateless components. In a serverless FaaS, functions are scaled for you. You don't have to do anything to handle a peak in web traffic. You don't have to do anything to handle a peak of msgs in your MQ.
In k8s, you still have to go and fumble around with CPU/memory limits and better get it right. k8s also doesn't scale your containers based on the msgs in your MQ out of the box. You have to build and run that service yourself (or ask GCP to whitelist you should you be running their MQ https://cloud.google.com/compute/docs/autoscaler/scaling-que... ). AWS Lambda had that since 2015...
Isn't that what the JVM/wasm solved?
And yet finding people who can reliably install K8s from scratch, who understand what's going on under the hood, remains remarkably close to 0.
How many people can, within a few hours, tell you how Kubernetes runs DNS, and how it routes packets between containers by default? How do you run an integrated DNS which uses, say, my_service.service.my_namespace instead of my_service.my_namespace?
I've found that most installs of k8s have been made using defaults, using tooling that Google has provided. We hired one such administrator, but when asked anything outside of how to run kubectl, they just shrugged and said "it never came up".
The codebase is vast, complicated, and there are few experts who live outside of Google. And it's getting more vast, more complicated on a quarterly basis.
It bothers me how far operations has gone from "providing reliable systems on which we run software" to "offload work onto the developer at any cost".
</rant>
I realize that a lot of this is because of scarcity. The good devops folks (i.e. those who are both competent generalist sysadmins and competent generalist programmers) are few and expensive. That makes pre-packaged "full stack" solutions like GAE, Kubernetes, and Fargate very appealing to leadership.
"You don't need an operations department to act as a huge drain on your revenue, just re-use your developers" holds a lot of appeal for those high up in the food chain. It's even initially appealing to developers! But in the end, it makes as much sense as re-using your developers to do customer service.
I've abandoned operations as a career path and have now gone into product management, but I was an operations person for more than 12 years. In that time frame I learned very quickly that upper management considered the operations teams to be "system janitors" and that developers considered operations engineers to be their inferiors. The "move fast and break things" attitude is great sometimes, except it gives license to shortsightedness.
The reality is that operations is not a specialized skillset, in fact it's a generalized skillset made up of being a specialist in multiple facets of complex systems. There's simply not that many people out there who have that level of knowledge and understanding, and the industry has both perpetuated this problem by treating operations people terribly and worked around this problem by focusing on building stacks that require minimal operational overhead. Any good operations person could have been a software developer, but wanted to get beneath the abstraction layers. Instead, we get treated worse, paid less, and have less job demand despite being more competent. Most of the best ops people I've worked with ended up either leaving ops entirely, like myself, or becoming software developers to get a pay bump.
Luckily I got to work for a few decent companies along the way in my career that treated me well and I made a lot of life-long friendships with very smart people as well. So don't read the above as some deep complaint. It's just an observation of the reality that the incentives aren't there for smart and talented people to invest their energy in operations. I advise most of the young people passed my way to become software developers. They'll have more autonomy, get paid more, have higher job demand, and get treated better in general.
Operations is the highly-skilled sucker who is awakened at 3am everyday and never paid overtime. Don't be that guy.
But been working as devops for 3 years now. I like it a lot. Especially automation part. What advice would you give me ??
Being on AWS or Azure or Microsoft doesn't shield you from these needs.
You need much more than Kubernetes: a secure (!) container registry, a container build system, deployment, log management, metrics...
It's fun to set up k8s from scratch, but there's little business value in reinventing the wheel all over again. Just like you wouldn't build your own Linux distro, you shouldn't do it with Kubernetes.
I've seen startups waste SO much time reinventing basic infrastructure instead of focusing on their product.
Not to mention, the costs for PaaS providers don't scale up well (if they can even handle the load). They're great for startups on VC, but deadly for companies who want positive cash flow.
Is it just the container devs dont know routing ?
https://kubernetes.io/docs/concepts/cluster-administration/n...
E.g. on AWS you might have all of a node's pod IPs on a bridge interface, then you talk to pods on other nodes thanks to VPC route table entries that the AWS cloud provider manages. NAT happens only when talking to the outside world or for traffic to Amazon DNS servers, which don't like source IP addresses other than those from the subnet they live in.
Personally, I'd rather just bring on ipv6. But, in my case, we don't have enough people who understand ipv6 (and it's barely supported in AWS) to use it ourselves.
Building containers that contain entire operating system gives no wins. In fact it add additional layer that will create issues, will break in a different way, etc.
The current love of the modern orchestration system by the management is similar to mid-nineties love of the "compute management packages" running on SGI that showed one "flying" though from one server to the other.
What is up with this? The last time I tried to learn kubernetes I couldn't find any information about how to set it up. Just some set up tools from google. I guess it is still like this? Is there really no one running kubernetes infrastructure with config management or anything?
I'm one of the most frequent commenters on #kubernetes-users so I'm very aware of the questions and issues that come in from new k8s users and I'd say an absolutely massive majority of the users are running in baremetal via kubeadm/kops/etc. Typically on AWS (NOT EKS). The #gke channel is literally 1/10th the size of the #kubernetes-users channel.
If you have questions about k8s post in #kubernetes-users. The community is extremely helpful.
A LOT of people deploy K8s clusters via Terraform/Ansible, as well.
Why are professionals who know k8s back and forth less common? 2 years ago k8s was 1.1 and we had no idea where the market was going and if it would take off like it did. It takes time to build up the community and expertise. There are a LOT of very experienced k8s users nowadays whereas there were not 2 years ago. Finding someone with 2+ years of k8s experience who isn't a Xoogler is fairly rare right now because 2 years ago it wasn't the market behemoth that it is right now. I don't work with Google but I just happened to get involved with k8s almost 3 years ago. We are out there.
If you can't find an answer ping me @mikej and I'll try to get you going in the right direction.
For example, the sidecar in a sidecar pattern might fail while the application is running and the system can get stuck in weird states. The developer still needs to understand fundamentally how the system works.
Eschewing deeper knowledge just because it is easy to use is trap in this case. While the article compares Kubernetes to JVM, Kubernetes can fail in a lot more hard to debug ways than the JVM right now. I don't know if this semantic gap between distributed systems like Kubernetes and monolithic systems like JVM can ever be bridged.
I would extend this to cloud as well. The more prevalent cloud becomes, the more ignorant developers become. It's like: I have Mathematica license, who cares how to calculate function derivative?
We are bound to fragment every sector into sub-niche where specialists in functions, general programming and infrastructure resources cooperate on their boundaries without being able to quite understand what the others are doing.
I would not say distributed systems are easy to build or use. I think Kubernetes makes distributed systems _easier_ but definitely not easy in general, or at scale. Just easier than doing it all by hand/manually.
But I'm not sure I understand the backlash. Once you've built your application and it's been packaged (containerized) and deployed why would anyone care how its run. Also running a container in production and orchestration seem to be conflated somewhat in this thread and the use cases are very different.
You can think of Kubernetes as an Automated SysAdmin . This is a bit reductive I know but it is useful to think of this way. You ask the sysadmin to run something for you and they tell you how to package it (tgz, war, zip etc) and they run it for you on hardware.
The level of engagement that a dev has with getting his app running on hardware is no different to that of dealing with a sysadmin and with the admin requesting that your app is packagedin a container.
Kubernetes out of the box will give you most of this functionality as long as you keep state outside of the cluster. There are also options on how to make the experience smoother. There also these tools to help too:
* Openshift * Kubernetes + Rancher * Mesos
If you need orchestration and scheduling. I am a little perplexed.
But I will mention that we're aware of some consulting organizations that are requiring that new employees take the exam after they are hired, as it gives both the engineer and their manager confidence in their understanding.
The exam has only been around for 5 months, but it's already gone through 3 versions and is based now on K8s 1.9. Also note that it's a proctored, online exam where you configure 7 clusters over the course of 3 hours. There's no multiple choice.
Registered nurses have been around for decades, so we have a long way to go to catch up their recognizability. But we do see the Certified Kubernetes Admninistrator as a core building block for the cloud native ecosystem.
While I don't think having a cert would help you get hired necessarily, it would probably influence a decision to get an interview. What really matters is if you know how to do real-world operational tasks with the knowledge, which will show up if your technical interviewers know k8s. If you are the first person they are hiring at the company to begin their k8s project, then you might have a real advantage with a cert.
Personally, I've never been one to give undue respect to many of the certs on the basis of having them alone, but it can depend on where you interview. Some places love certs.
