The author suggests GASNet, Charm++, and several other things that would be comparable to MPI.
The examples given in Spark and Chapel show how much better it can be. Such methods deserve much more investment. The bad thing is that the DOE labs are clinging so hard to their existing tools when they're in the best position to invest in new ones via huge grants they have. They built supercomputing before anyone heard of the cloud. Their wisdom combined with today's datacenter engineers could result in anything from great leaps in efficiency to something revolutionary. Yet, they act elitist and value backward compatibility over everything else. Starting to look like those mainframe programmers squeezing every ounce of productivity they can out of COBOL.
I think the change will have to come from outside. A group that works both sides with funding to do new projects. They either (a) improve MPI usage for new workloads or (b) get the alternatives up to MPI performance for HPC workloads. Then, they open the tool up to all kinds of new projects in both HPC and cloud-centered research (esp cloud compute clusters). Maybe the two sides will accidentally start sharing with each other when contributing to the same projects. Not much hope past that.
Minor historical pontification on my side: The term 'cloud' is only the current term for an old idea. Before cloud there was 'grid'. Before that was the Amoeba OS and other distributed OSes. Long before that, a 1960s hope for Multics was that it would be:
> ... a utility like a power company or water company. This view is independent of the existence or non-existence of remote consoles. The implications of such a view are several. A utility must be dependable, more dependable than existing hardware of commercial general-purpose computers. A utility, by its nature, must provide service on demand, without advance notice in most cases. A utility must provide small amounts of service to small users, large amounts to large users, within very wide limits. ... http://www.multicians.org/fjcc3.html
That's shortly after the CDC 6600, and before the DOE really got into the supercomputing business.
In any case, I don't know the politics of supercomputing, and my experience in that field is even older than yours. When I look at current clouds systems, and try them out, I throw my hands up, because they don't handle the types of architectures I use in my research. I do compute-heavy work on medium-sized data, with algorithms that works best with stateful nodes in a boss/worker system. Luckily for me, I don't need low latency, so building something on top of ZeroMQ is good enough for what I want, and easy to do.
