Kubernetes is designed for running Application Container images (e.g. docker, oci, appc) and in this case those containers are running not as Linux containers on a shared Kernel but inside of a VM with Xen/Hypernetes. Another implementation of this "app container in a VM" concept is Clear Containers which is available in rkt[1] and can, like Hyper, run the application container image formats (docker/appc).
This is all a bit confusing so let me know if this helps :)
The broad approach isn't new (https://clearlinux.org/features/clear-containers describes Intel's Clear Containers work which allows you to use rkt to run containers in individual lightweight VMs), but the trick in this case is to perform the isolation at the pod level (ie, a collection of functionally related containers) rather than the individual container level.
(Edit: Apparently the Docker support for Clear Containers hasn't gone upstream, so dropped the reference to it)
You can use OCI Runtime bundles with rkt if you use the oci2aci[1] tool and rkt will be a full OCI runtime once the OCI Runtime and Image spec mature. We can use help to get these OCI projects[2][3] to v1.0 if you can spare cycles!
[1] https://github.com/huawei-openlab/oci2aci
See https://blog.docker.com/2016/04/docker-engine-1-11-runc/
Make sense?
What I'd rather see is an allocation layer for physical resources that just cordons off the whole machine (physical or virtual) by tenant as soon as previous tenant resources have been fully consumed, then reclaims hosts after usage subsides. So as a provider I still only have one cluster to manage, but as a consumer I still don't worry about another layer of abstraction slowing things down or pre-allocating resources.
Currently, if you want to resell computing resources, you need to rent or buy a dedicated server, and run a hypervisor on it.
Containers enable a new class of reselling computing resources. Because you can run a container within a VM, you can resell computing capacity on a VM.
I think we are going to see another abstraction on top of "the cloud," due to this additional layer of reselling (new russian doll on the inside, new doll on the outside).
The physical abstraction is:
Datacenter > Floor Space > Server Rack > Server
The virtual abstraction is:
Server > VM > Container > Kubernetes|MESOS|...
Virtual is a 1:1 inverse of physical. Next step is datacenter interconnections (i.e. multihost kubernetes or whatever flavor of the month IaaS software people use).
HW -> EC2 scheduler -> VM -> You -> Container scheduler -> Container -> App
There should not be two schedulers if container is secure for multi-tenancy.
This means that by default, anyone can do anything they want with your cluster.
There are no warnings, no suggestions that turning on the much better TLS based authentication would be a good idea (or even how to do it), no nothing.
Be very careful with Kubernetes.
[0]: http://kubernetes.io/docs/getting-started-guides/ubuntu/
Sometimes you have to take a stance on these types of things, as we have done with the CoreOS + Kubernetes community guides [0]. The guides are open source, but full TLS, passing conformance tests, etc is required for contribution.
(I work at CoreOS)
[0]: https://coreos.com/kubernetes/docs/latest/#installation
One of the main benefits I have now is that if I run a number of containers that all take various amounts of memory, I can just throw them on and they share memory amongst each other quite efficiently. If I have to make a static allocation of memory for a VM, I'll typically choose a conservative memory number, and usually under-utilize the machine, wasting a lot of memory per-instance. Not so bad since I chose per-pod, but still an issue.
As it happens, this same issue is why I'm leaning towards lightweight native applications these days instead of an aggressive greedy virtual machine that grabs a bunch of heap. Golang/Rust in particular.
And the scheduler will need the mem size (not right now, but inevitable)
- containers - openstack - multitenancy
