* Matei Zaharia is the creator of Apache Spark
Given a set of cheap SSDs and a good I/O scheduler, very few workloads should bottleneck on disk because the disk has more available bandwidth than the network. If you are bottlenecking on disk, it usually means the I/O scheduling is poor. The operating system I/O scheduler is quite poor for many use cases, hence why I/O intensive apps write their own.
Network can be inexpensively and easily saturated at 10 GbE these days, even when doing something ugly like parsing JSON streams. Unless there is a bottleneck elsewhere in the system, like memory bandwidth or occasionally CPU, this is where I typically see bottlenecks in real systems. However, switch fabrics do not scale infinitely even if you know what you are doing, so there is that, and bandwidth does not always scale linearly (though things like graph analysis are much closer to effectively linear in sophisticated implementations than you see using naive algorithms).
Data motion is always the bottleneck. Right now, moving from RAM to CPU or from machine to machine is almost always the bottleneck if you are doing everything else right. Many popular software designs and implementations for "big data" have many other bottlenecks that are not strictly necessary, so that throws off expectations e.g. poor I/O scheduling saturating disk or poor JSON parsing burning up CPU or poor use of threading wasting a lot of CPU time.
At some point, a larger cluster will be more bandwidth-constrained because of oversubscription, but given the network utilizations we saw (<5% at the median in Figure 5), a cluster would have to have pretty high oversubscription for the network to become the bottleneck.
The one caveat here is, for example, matrix workloads, where the data sent over the network increases with the number of machines.
1. Obviously, it costs more to get a bigger cluster.
2. You might not be able to get as big of a cluster as you wanted, since "cloud" is not an infinite resource.
3. Some workloads are not embarrassing parallel. For some (graph, matrix, join), more parallelism -> more communication.
