I am developer, but mostly work on desktop apps, or embedded devices or lataly on some MVC applications. But reading things like
"A Container-Native Stack for Modern Applications and Operations Increase development velocity while simplifying operations."
I have no idea what should I imagine and what is it good for...
any good introduction or explanation into what is it they actually do?
In the ancient past, like when I'm from, you'd write up a few different bash scripts to help you provision each server type. But setting this all up, you'd still have to run around and create 20 servers and provision them into one of 5 different types, etc.
Then there's chef/puppet, which takes your bash script and makes it a little more maintainable. But there are still issues: huge divide between dev/prod environments, and adding 5 new nodes ASAP is still tedious.
Now you have cloud and container orchestration. Containers are like the git repos of the server world. You build a container to run each of your apps (nginx, redis, etc), configure each once (like coding and committing), and then they work identically on dev and prod after you launch them (you can clone/pull onto hardware). And what's more, since a container image is pre-built, it launches on metal in a matter of seconds, not minutes. All the apt-get install crap was done at image build time, not container launch time.
Things are a lot easier now, but you still have a problem. You're scaling to 30, maybe 50 different servers running 6 or 7 different services. More and more you want to treat your hardware as a generic compute cloud, but you can't escape that, even with docker, your servers have identities and personalities. You still need to sit and think about which of your 50 servers to launch a container on, and make sure it's under the correct load balancer, etc.
That's where Kubernetes steps in; it's a level of abstraction higher than docker, and works at the cluster level. You define services around your docker containers, and let Kubernetes initialize the hardware, and abstract it away into a giant compute cloud, and then all you have to do is tell kubernetes to scale a certain service up and down, and it automatically figures out which servers to take the action on and modifies your load balancer for that service accordingly.
At the scale of "a few servers", Kubernetes doesn't help much. At the scale of dozens or hundreds, it definitely does. "Orchestration" isn't just a buzzword, it's the correct term here; all those containers and services and pieces of hardware DO need to be wrangled. In the past it was a full time sysadmin job, now it's just a Kubernetes or Fleet config file.
Disclosure: I'm currently writing a book on Docker. Disclaimer: I have not had my coffee yet.
Edit: Since someone asked, I'm writing a book called "Complete Docker" which will be published by Apress. I don't know the exact pub date that Apress will launch it on, but I expect it'll be available in October.
Now I wonder.. how many projects actually needs these kind of solution when even StackOverflow can do without it (they are in the range of few servers)? I would imagine it would be only few top popular web apps/services, but by popularity of these posts it looks like it is probably a lot more...
Does it make things better, though? Yes. Yes it does.
Kubernetes was designed at Google where they reaaaaally feel their scale problems day to day.
I see Docker/Kube/CoreOS/ etc as the natural evolution of where we were already going. Bash -> puppet -> vagrant -> docker -> kubernetes. Less abstract to more abstract.
So it's actually "only" an incremental evolution in terms of managing the server ecosystem. But it's a revolutionary improvement in how we think about server ecosystems, which is why many people struggle with Docker et al at first; it's a brand new mental model.
Really, why should cost anything to have basic, low-traffic, site on the internet today? That's the quetion I ask myself.
There's also other aspects, like being able to spin up the newest PostgreSQL within minutes, and when you're done you rm the container and image and have no trace and/or side effects.
There's some pretty useful multi-container configurations out there too, like a dockerized gitlab environment: https://github.com/sameersbn/docker-gitlab
Something I never understood with containers: where do they store persistent data, e.g. MySQL's /var/lib/mysql - and how does upgrading work, i.e. when the apt-get postinstall script runs transformation on the persistent data, how is the transformation applied to the "clones"?
You don't upgrade a running container. Imagine that containers are immutable; to launch a new, upgraded version, you re-build the image in your build/dev environment, and re-launch the image into production. If you're using a volume, you get to use the same backing data.
It's rare that apt-get postinstall will affect any data that you would persist -- app-specific data you'd keep in the image/container, and mysql data for instance you don't really want apt touching anyway. But if a data migration is necessary you'd either manage it with a "utility container" (image that's designed to run a script then stop, rather than run and keep running).
Right now the first several chapters serve as a "I'm a programmer but know nothing about Docker" guide, the mid of the book dives into Docker-specifics (how exactly volumes work, how Docker networking works, etc), and then it graduates into the Docker Ecosystem, dedicating some time to covering tools like Kubernetes, CoreOS, Fleet, Amazon ECS.
The latter portion of the book is a number of recipes: how to get a Ghost blog running behind an nginx proxy; how to launch WordPress with MariaDB; how to launch an ELK stack, etc.
And hopefully your book will mention SmartOS and Triton and zones providing full isolation for Docker.
"At the scale of "a few servers", Kubernetes doesn't help much. At the scale of dozens or hundreds, it definitely does."
So now I know I don't need to learn much about Kubernetes unless I have "dozens or hundreds" of servers to manage.
One thing Kubernetes IS helpful with at small scales: portability. If you're fully Kubernetes/Docker, then you won't get locked into (eg) AWS's ecosystem. It's relatively easy to pick up an entire Kubernetes cluster and move it from AWS to DigitalOcean to private hardware.
So even if you're at small scale today but want to design for portability, I'd definitely look into Kubernetes.
Now, think about it from a reductionistic engineering perspective. What do I really want this server to do? Well, it accepts TCP connections, parses a request to figure out which file (at the time, it was all files) to server, sticks an HTTP header on the file, and shoves it down the socket.
This task is so simple that a skilled network programmer can nowadays literally bash together a 1990s-level static HTTP server in an day, with nothing but a socket library and some basic string handling. (It may not be great and it probably is insecure, but, well... see also "1990s web server"....) The code to do this is perhaps in the dozens or hundreds of kilobytes.
But that's not what I had. I have a full computer that physically needs to live somewhere. It has hardware ethernet and hardware graphics cards and a physical monitor and a power supply and RAM and, basically, hardware hardware hardware, the failure of any one of which means the system is either difficult to change or outright down. I have an entire Windows operating system, which even in the 1990s was hundreds of megabytes of code, endless code. Code for a windowing system, for pete's sake. Code for the audio subsystem. Code for accessing the hard drive. Code for access code that accesses code. Code code code code code, a bug in any one of which means the system may be down or insecure. My website, which at the time was quite likely in single-digit megabytes in size, was a tiny directory lost in a sea of files on the hard drive.
Over the past 20 years, the commodity hardware world [1] has been slicing away at the fact that several dozen kilobytes of code are being accompanied by hundreds of megabytes of support and literal pounds of physical hardware. Hardware went first with VMs. VMs got lighter and lighter. Lightweight hypervisor solutions sliced away at the heaviness of the VM. Containers slice away at the OS. Things like Kubernates slice away at the idea of a container living somewhere physically.
We're trying to free that several dozen kilobytes of code to be just several dozen kilobytes of code, as flexible and easy-to-deploy as several dozen kilobytes should be, if you weren't mired in the world of hardware and OSes and code and strong physical connections.
(Data storage is more complicated, but in a lot of ways, the same principles are in play.)
Operationally, containers are very exciting. However, in terms of "magic technologies I don't understand", I don't think they're worth stressing about "getting old" or anything. It's mostly "just" a big pile of practical considerations in trying to not just build that world, but in some sense also undo decades of grinding-in of the physical world to our operational considerations. If you want worry about getting old and out-of-date, worry about that let you do something that you couldn't before, like GPU programming or deep learning.
[1]: Which must be specified because mainframes beat us all here decades ago, in a lot of ways.
Cloud orchestration: the logic for how your platform provisions and launches instances (think ec2) into your fleet of resources.
Container orchestration: the logic for how your applications (containers) are distributed across your fleet of instances
Kubernetes: Software that runs on your fleet and manages how/where/when container orchestration happens. There are many players in this space such as mesosphere, swarm, ecs and so on.
1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.
2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.
3. Anything invented after you're thirty-five is against the natural order of things.
- Douglas Adams
i'd argue that this situation is if you're a developer at a large company or enterprise. developing can be a nightmare because testing and/or deploying is so difficult. sometimes this is by design, sometimes by virtue of incompetence. other times people want to change it, they just don't know how.
tools like this claim they can change this. whether that's true or not. well, that depends on more than just what tools you use but it is part of the equation.
[1] https://www.joyent.com/blog/samsung-acquires-joyent-a-ctos-p...
Please, please continue to develop your public cloud offerings. Having options other that the myopic, me-too, feature-matching, monoculture that is AWS/GCE/Azure is incredibly important.
That said, for my use profile, you guys need to work on your price competitiveness. Hopefully Samsung will inject the necessary cash for economies-of-scale.
I'm sure they will. They are a top contributor to open source:
Wiki: "In journalism, lede is an alternative spelling for lead paragraph."
We use Joyent for nodejs microservices, chiefly any server side DOM manipulations (using jsdom), and it's been solid! Currently evaluating docker container management and will give Triton a go...
Just saying, there's a bit of bias that's probably worth recognizing.
I don't remember the exact Korean wording, but a common phrase said by Samsung engineers (under their breath) in Korea basically translates to "I'll do what you say, but it's only because I'm at <name of a Japanese labor camp that I forget now>." My time there really made me appreciate how much better we have it here!
Can you explain the cultural aspects (and management shortcomings) in more detail for those of us who haven't experienced Korea? Thanks.
In general, the culture is very hierarchical (respecting elders) thus seniority and age play a big factor in leadership. I would assume that this makes it very easy for mediocre (but good enough to survive) management to stay in positions far too long. Asian culture, but Koreans especially, can be masochistic1 and martyr2 like in their methods. In some ways this creates a tremendous amount of excellence, but it also creates soldiers who may not necessarily be incentivized to innovate, and those who innovate, out of cultural respect will tend to not dare "disrespect" seniority. Being a good soldier makes for career advancements.
Management can also claim the failures of the company as their own personal failures, rather than the gray collective failure of a group(e.g. japanese ceo's claiming company failure as their own), thus employees kind of get absolved of blame.
Work in Korea also tends to be a live at the office kind of affair. It's not uncommon for the white collar folk to go into work at 7a, and stay until 7-9p. Then go drinking with coworkers until midnight, and rinse and repeat. While this makes for very strong bonds amongst coworking peers and better teams, I would imagine that it also takes a toll on the mental capacity of the workforces.
Mainly, I wouldn't be surprised if work in Korea tends to be death marches all the time based on the martyrdome syndrome and masochistic syndrome for career advancement.
1 - http://askakorean.blogspot.com/2010/01/koreans-english-acqui... makes 30k vocab flash cards to learn english 2 - https://www.youtube.com/watch?v=S-yD4OfY8cM boyfriend gives up both eyes for girlfriend he blinded. Why not keep 1 eye?
Basic courtesy and being polite are pretty much universal, so try to be humble and keep an open mind and observe. Older people often are more experienced and savvy about how to handle something so being deferential to them is a no-brainer. You'll spot the incompetent ones soon enough and so continue being polite but ignoring their guidance.
One thing that drove me nuts but took a little while to figure out was that people would almost never say no to a request. I would ask if something was possible and they would pretty much always say yes, not wanting to disappoint, but I would have been better off hearing a 'no, sorry'. I wanted a realistic answer and theirs were often far too optimistic.
Me: "Mr Son, can we make it to Osan in 30 minutes?"
Him: "Yes" (always yes)
An hour later we show up late to another hospital with a very impatient air crew idling and waiting for our patient transfer. We would have all been better off hearing a "No, it takes an hour". Just one small example but this scenario repeated frequently.
I learned that when speaking to a group or doing a presentation it was customary to apologize for your inexperience on the subject and promise to do your best. Tim Berners-Lee could start a talk with "I'm sorry I don't know much about the World Wide Web..." for example and Westerners would laugh but the locals would be like "He's off to a good start".
Understanding this helped me get more realistic answers though. Even if I thought I knew an answer I would keep the question more open-ended to solicit a better response.
Me: "Mr Son, I'm sorry but I forget how long it takes to get to Osan, can you remember?" (said even after making the trip solo dozens of times in often difficult and stressful conditions).
Him: "Ah yes, there is new road construction halfway there so I'll teach you an alternate route to bypass it. We'll be there in about 40 minutes".
Had I not learned to ask a better question, I never would have learned his faster alternate route.
Korea was a lot of fun and I really enjoyed hanging out with the older Koreans who worked on our base, they were a constant source of good info once learned how to talk to them. I also had a 'ration card' entitling me to buy the occasional bottle of Crown Royal on base and give them, which always helped!
Whenever someone in HQ tells you to do something, you do it, and if not, you get punished, whether that's through decreased compensation or they fire you. You can be the engineer that tells managers that you can't draw parallel lines that intersect (metaphorically speaking), but they don't want to hear it unless the competitor proves them wrong with sales results - they are notorious for not innovating, but copying competitors. Samsung runs on waterfall from top to bottom. Oh, and if they tell you to do something early in the morning, you jump.
Samsung also tends to mainly promote Koreans - if you're not Korean and want to get ahead, you'd best look elsewhere because you're not likely to accomplish it. I hear this also applies to their American divisions as well, but I do not know anyone who has worked for them here. The family who owns the company has had some history with corruption in Korea as well, so this isn't too surprising.
More on the cultural side, Koreans can be very rude by western standards - it also is demanding on expectations. You're often expected to dress a certain way (this part is no different from most of the world, but disregarding norms is probably more harshly regarded in Korean culture than most), as well as join bosses & other co-workers going out to drink and doing karaoke at night. Deferring to father-like/older figures is built into the language with even how one addresses another, and that bleeds into how companies operate (& people treat each other - even being one month older can come into play unless you have leverage of some sorts). There are also built in opinions on processes, from parenting to schooling to work. Koreans tend to also be quite xenophobic/racist, even against Koreans who cannot speak Korean (maybe even moreso towards Koreans who cannot speak the language). Society sort of disowns those who are not immersed in the culture. Non-Asian foreigners are special, but they are excluded from much of anything of consequence (probably to avoid stirring the pot with other nations).
There's probably a lot more I'm missing here, but tl;dr if you aren't an extremely mentally tough person, working for a Korean company in Korea can crush you thoroughly. It is a different world, and people who do so from western society are left without a support base.
That's Samsung proper. Not their incubator setup, which is different. Companies like Mapzen were bought by Samsung, but have a completely different culture.
This is good news for node.js since what this means is more money gets poured into node.js development.
For example Microsoft pouring money into JS seems to have lead directly to Typescript.
Later, I visited their HQ in Korea and met some of their Indian and Korean development teams. They were competent but the work culture was toxic. People were expected to attend 11PM daily review meetings where the issues list was painfully re-read pointlessly and nothing at all was achieved. Koreans were expected to live in numbered company compounds, catch a numbered company bus and ascend to a numbered floor in a numbered company skyscraper after an airport security style physical check. Smoking allowed strictly twice per day in a designated corner of the compound. All movement is tracked. Psychotic Confucian management style. I honestly thought to myself: if this is a vision of the future, I want no part of it.
Samsung Semiconductor is happy to supply chips to Apple even though they're the biggest rival of Samsung Telecommunications the builds the Galaxy phones. Conversely Samsung Telecommunications may or may not pick Samsung chips for their Galaxy phones, it doesn't really matter.
So I guess it depends to what division they will be attached. If they're attached directly to the group, they may actually keep complete freedom and just have a different owner.
https://www.glassdoor.com/Reviews/Samsung-Research-America-R...
This could just be clashes between US and Korean management culture, or perhaps just some instability in a new research lab.
Compared to the kind of stuff that Microsoft, Apple have done in the past in the software world.
And I do not see how this is bad, maybe Samsung is trying to learn why 'Western' engineering is so much superior to 'Asian' engineering by this acquisition ?
Microsoft is a software company. Samsung is not. If you think they are, then you probably haven't used one of their bloated, buggy phones that they stop updating way too soon.
jk.
There's a reason Samsung seems to be not that great as a software shop. I wrote up a long post on HN almost 2 years ago (God, it's been 2 years!!). I'm reposting the link here.
We were offered either one or three free years of hosting on a Joyent SmartMachine, depending on what your initial investment was. After this free trial, you would be converted to a regular customer.
Adding my voice to the bitter-parade. Joyent wouldn't be where they are today without us early bootstrap investors. It was basically a Kickstarter campaign before Kickstarter existed. Not only do I feel like I failed to receive my early investor benefits, but WORSE off was how the whole situation was handled. We had to beg and plead to get our data back. The company essentially went dark to all of their oldest, most loyal customers.
I totally take the "lifetime" deal with a grain of salt but it wasn't like the company folded because it ran out of money or wasn't successful. It was more like we got successful so we forgot about the little people.
Congrats to all my former colleagues who are absolutely amazing at their jobs and wonderful people to work with. Samsung looks like a very good match. Hope the transition goes well.
So that's how I saw it progressing. Samsung was looking into how to get in on the Internet of Things, stumbled across Manta, and it went from there.
If you want to store objects in a distributed way, why not just use something like RethinkDB, which has proxying/sharding etc. built in? RethinkDB even has a way to execute code with 'js'.
Or for serverless processing, why not AWS lambda or the crop of clones?
[1] https://github.com/joyent/smartos-live
[2] https://github.com/joyent/illumos-joyent
[3] https://github.com/joyent/manta
[4] http://www.theregister.co.uk/2013/06/25/joyent_object_store/
Say what? SmartOS is based on illumos, the only free, open source SVR4 UNIX out there.
Get the code: https://github.com/joyent/illumos-joyent
Build it! https://wiki.smartos.org/display/DOC/Building+SmartOS+on+Sma...
Modify it: https://wiki.smartos.org/display/DOC/SmartOS+Developers+Guid...
Data locality.
I'm not sure where this will take things with Samsung, likely backend services to support their own "store" on their devices separate from Android. If it's internalized, I only hope they release more of their infrastracture tooling, which is very cool to say the least.
- lx-branded zones and window's new layer are both doing system call translation---you're post doesn't contradict this but doesn't make it clear either.
- At some point lx-branded zones where equally untested.
- "Desktop class" is misleading because NT was designed for servers. Now yes Windows as a whole is hamstrung by a business model putting desktops and backwards-comparability first, but by definition the Linux layer can ignore a lot of Win32 precedent.
- Yes Illumos is Unix and NT isn't, but the difficulty in translating linux systems calls lies not with the ones backing posix-interfaces, but the Linux-specific ones. (Joyent devs can give you some great horror stories here.) It may be a bit harder with NT but it was hard already.
I'm ignorant on why Joyent is so superior to Azure. Can you give some reasons? As an outsider, Joyent looks like a small company with unproven (large, successful customers are the main criteria) products that cost more than the comparable solution from AWS, GC, Azure.
It's a good product fit, but the fit is so good that Microsoft basically built a more-complete competitor in the form of Azure. They built Joyent instead of buying it.
LinkedIn was different because Microsoft could never build LinkedIn itself.
[1] https://azure.microsoft.com/en-us/services/app-service/web/
edit: I should have been more specific.. when an open source project is under a company's wing and it gets acquired, you don't know what can happen, even under MIT. Look at express recently. Since it's the under the Node Foundation now, this is not a big deal. Had it happened a short time ago, there may have been further turmoil in the community.
So actually all of Cantrill's worries with Oracle exist with Node.js's license. I am sad when I see large free software projects licensed under MIT, we are doing ourselves a disservice by allowing proprietary software to take advantage of the free software we've developed.
[1] It's part of his Fork Yeah! talk about the fork that is illumos.
On the other hand, I have nothing against CDDL. I think the incompatibility of CDDL with the GPL is what saved the illumos community (even if inadvertently) from having their technology cannibalized by GNU and Linux. I'm happy that BSD is benefitting from illumos (and vice versa), but I'm even more happy that Linux isn't. <--- I do not represent anyone but myself, the position expressed here is my own and not of anyone else.
"By bringing these two companies together we are creating the opportunity to develop and bring to market vertically integrated mobile and IoT services and solutions that deliver extraordinary simplicity and value to our customers."
Given how well React Native tends to work, and I have to be honest, I like the ecosystem structure better than most apps I've worked on in general, I can see it actually working out better in many cases.
Prob not great for more advanced messenger functionality but Face Slim (https://github.com/indywidualny/FaceSlim) does notifications and simple/group chats fairly well.
- AOT compilation for a subset of JS: http://www.eecs.berkeley.edu/Pubs/TechRpts/2015/EECS-2015-13...
- "A JavaScript engine for Internet of Things": http://samsung.github.io/jerryscript/
Its corporate culture only allows the most cunning, politically savvy person to stay alive and move up the rank, and thus most executives (all if I limit it to small sample of executives I've personally met) fit that model.
And shit literally flows downwards, where goals/promises set by them would be pushed downwards and engineers have to take the burden.
It doesn't help that Korean society is very hierarchal and based on Confucius principles, where you don't usually challenge older persons and/or someone higher in the rank. This is one example that describes serious problem - http://thediplomat.com/2013/07/asiana-airlines-crash-a-cockp....
For those of you who are intrigued and have time, I suggest watching Misaeng with English subtitles (https://www.viki.com/tv/20812c-incomplete-life). Samsung isn't as bad, but the same hierarchy, verbal abuse, social dynamics, and strict rules on paper format exist.
The best outcome would be if they leave Joyent's management and culture alone. But I doubt it.
I also have the first-hand experience of their applying the same "consumer electronics" mentality to completely different business which required high-touch sales.
There is no denying success of Samsung - multi-billion, international corporation. However, Samsung is only good at generating quality hardware products at mass scale. There have not been success in any sort of software and services. Perhaps they are trying to expand beyond their strengths, and I applaud that effort and they actually do need it, since it's only matter of time Chinese companies will catch up and produce as quality products as Samsung, as Samsung did to Sony. I hope it bears fruits. I hope they can allow Joyent to succeed and thrive, and learn from that.
I will see what happens next few years.
"Samsung will immediately benefit from having direct access to Joyent’s technology, leadership and talent. Likewise, Joyent will be able to take advantage of Samsung’s scale of business, global footprint, financial muscle and its brand power."
Their fears have come true (and now it's even happening to Apple). Hardware is an incredibly difficult, risky business, and differentiation is now in the services/software arena. That's why Samsung, in particular, needs this acquisition (and others like it).
Disclaimer: I'm not involved in that space, so maybe they've started?
I don't expect to see Samsung take on GCE, Azure or AWS directly though... and would be really unsure if I were a Joyent customer right now.
That said, since SmartOS is based on illumos, and illumos is designed from the ground up to be portable, running SmartOS on ARM would be cool, since it would open the doors for illumos on embedded devices.
I would so love me an illumos based, iPad Pro-like tablet...
It seems that NodeJS has moved out of Joyent. They have hosted container support that seems to run on solaris, which seems interesting, but a bit too much of buzz-wordy from their website.
I am not very familiar with this, so will be great if someone can explain a little bit. I read the comments around orchestration, but am more interested in Joyent's value proposition.
If anybody deserve it, it's you guys.
