I call these "weekend benchmarks" -- what you'd typically do when you have a block of free time, then spend time optimizing for said benchmark. Roll in on Monday with some staggering results, only to find one (or many) of your variables were off.
Did the author try multiple instances on each provider? VM tenancy is a bitch. (Think of how annoyed you get at the noise levels, when your neighbor in the next apartment throws a party)
Is the author's source benchmarking machine, a physical machine, or a virtualized guest? Does it have power savings turned off so that the process is running at 100% speed, instead of a variably-clocked down core?
Did the author enable or disable TCP time-wait recycle? So he doesn't bump into said ceiling when running such tests back to back?
Did the author run the tests back to back, or have a cool down period between tests?
Where was the author's network upstream located when he tried said tests? Were there any network issues at the time of the test? Would the author even be aware of them?
Your page you're testing against, does the same database call, which is presumably cached. Did he throw out the cached results? Can he identify the cached results?
Are we firing up ApacheBench with HTTP keepalives enabled? With parallel connections? How many parallel connections?
How many Apache servers (StartServers, MinSpareServers, etc)? Which httpd modules were enabled? Which httpd modules were disabled? Which PHP modules were enabled?
You're trying to benchmark CPU and I/O horsepower across three different platforms but doing it through this narrow "straw" which consists of your "independent server", your upstream, your upstream's upstream, your upstream's upstream's peering connection with Amazon/Linode/DigitalOcean, your web server and its PHP module, your application, and MySQL.
If you're rolling your eyes at this, then you shouldn't be doing weekend benchmarks.
I'll leave you with this as well:
I'm particularly fond of the quote "lead, follow, or get the hell out of the way", which is a bit harsh in this case because a lot of your advice is good. It could be framed in a more constructive way, though - there's some Comic Book Guy tone there in your comment.
My intended tone was not "don't try", but "try harder".
I've listed at least 5 ways to improve/normalize the testing, as well as linking to a document that does a pretty good job of explaining statistics (particularly, how programmers do a bad job of statistics; baselines for benchmarks; etc).
"At least he's out there trying" -- with this not-so-great benchmarking, the author has just effectively SHITTED on 2/3 companies that have gone to great lengths to build amazing infrastructure AND managed to spread his FUD around the web, to the point where it reached the HN front page -- and you want credit for trying?
Get the hell out of the way.
"At least we're doing something!" is a silly defense.
Another major flaw is taking results for a single instance type and implying that those apply to all instance sizes and each provider as a whole.
If you're going to do a benchmark at least pick something realistic like the m3.* types: http://aws.amazon.com/ec2/instance-types/instance-details/
At least the author had enough sense not to do the bench on a t1.micro
If you are working towards "best practices" on AWS, you should be running multi-region (who wants to be the one left holding the bag when US-EAST goes down again?). If you've done all the heavy lifting to enable yourself to run in "pods" across mutliple regions.
Well, if you can do that, why not treat Linode/DO/Rackspace as a separate region and deploy a "pod" of servers there?
At the end of a month you should have enough statistics that are directly applicable to your own app and your specific customers, as well as some real experience with the operational experience of dealing with the new provider.
For example, maybe one of the other providers has really fast machines and their major upstream provider has a great peering relationship with whatever test node you were using for these microbenchmarks, but perhaps those servers are really flaky and crash all the time, or perhaps the majority of your customers see really bad latency when hitting those servers? Maybe their API isn't just "immature", maybe it crashes a lot and they have bad customer service.
Those are the sorts of things you aren't going to figure out after simply running a few load tests. Anyhow, it just seems like something like this would be a lot more valuable than any amount of synthetic testing.
edit: typos
I'd much rather see a two or three synthetic benchmarks around harddrive throughput/latency, memory throughput/latency and CPU.
Why choose the fastest one? This would be the least accurate way to give an indication if the performance of AWS instances. A mean, or perhaps median depending on the skew, would be a better choice.
Unless you're saying that the benchmarks on the other servers are effectively cherry picked best results.
AFAICT, Linode doesn't offer that, and they probably won't. Amazon's been ahead in this arena for awhile, and will probably keep that lead for the forseeable future. EDIT: Apparently they do have an API which would cover a decent variety of use cases.
What's sad is the number of people that migrate over to Amazon because it's the done thing, without realizing what they're paying for (and that they're not utilizing the unique features of EC2).
While perhaps not entirely as mature and full featured as AWS's offering, Linode does offer an api with which you can script creation/initialization of test servers.
See https://www.linode.com/api/linode
They also offer something called StackScript, that to me looks like some kind of configuration management script. https://www.linode.com/stackscripts/
But they do not make it very easy to spin up / down new instances. Setting aside the API, they bill you for a full month for a new instance as soon as you create it. True, you get a pro-rated credit if you delete it earlier but this is awkward and I think it only pro-rates by the day.
And StackScripts kinda sorta work, but they are hard to write/debug and are not portable. It's a pretty weak offering.
Another important aspect is ability to easily transfer raw data into an analysis tool, like Red Shift. Google also excels here with their Compute Engine and Big Query.
While it is cool that AWS has a ton of services they offer, I do not like the vendor lockin that comes with AWS services. I think it is generally a better strategy to go with something OpenStack based. That way you can use your own hardware and dynamically provision new nodes in "the cloud" with companies like HP, RackSpace, and others. With modern day virtualization, running your own hardware can often be much cheaper and much more performant than AWS.
Digital Ocean does do metered billing, and their API is very sweet & simple.
Obviously the API isn't as mature - but it is going in the right direction
This absolutely should not be true, and in my experience it isn't true on Amazon -- you will neither be starved for resources if others do intensive things, nor will you enjoy riches if they are quiet: You will get what you are paying for.
I have a personal experience from Digital Ocean that is a bit different. Firstly let me say that I think they have a great service and compelling prices, but I set up a test server (the 2GB/2CPU variant) to trial leveraging it in the platform mix, as a solution that crosses host providers = awesomeness.
The IO performance I got was terrible, despite all of the talk about SSDs. Simple operations would stall, the CPU endlessly waiting on wa. I submitted a support ticket and quickly they toggled some priority flags and I started getting performance more along the the lines of expectations, but ultimately it seems like a classic case where single tenants can completely monopolize the platform, enjoying the entirety of the storage platform at the cost of everyone else. I'd rather that they cap consumption and do appropriate IO quanta allocations rather than leaving VMs starved.
And it really makes me concerned for the future -- do I have to constantly do benchmarks and analysis, hopping VMs just to find one that isn't an abomination? That isn't how these things are supposed to run.
I have heard anecdotally that it's not uncommon for large players to bulk-start instances and then kill all of the ones that don't have the latest hardware.
Update: I just looked at 4 random c1.xlarge instances we have running, and found 3 different types of underlying hardware:
1. Intel(R) Xeon(R) CPU E5410 @ 2.33GHz w/ 6144kb cache
2. Intel(R) Xeon(R) CPU E5506 @ 2.13GHz w/ 4096kb cache
3. Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz w/ 20480kb cache
Of these cores, the E5410 is from 2007, E5506 is from 2009 and the E5-2650 is a Sandy Bridge from last year.
They should still set things up so you get reasonable baseline performance, even in the high-contention case, rather than overselling the resources. But you can end up with quite a bit of performance variance in the upwards "more than your fair share" direction, especially for I/O, if your neighbors are quiet. If you're on a 32-guest machine where everyone else is idling on IRC or doing nothing, you get a whole disk to yourself; if everyone is doing random alternating reads and writes, you get a 1/32 of a thrashing disk's worst-case throughput. Usually you get something in between.
Regardless, we stick to AWS at work for the entire suite.
When it comes to my money though, DO is making very large strides imo
http://www.techempower.com/benchmarks/
Would be nice to see similar benchmarks using a framework that doesn't resemble a honeypot's tarpit.
You're more than welcome to help make laravel look better in these benchmarks. But unless you're running the PHP through HipHop, I doubt you're going to get anywhere near the statically-compiled platforms near the top.
We get around 50TB of data transfer per month from Linode, from having around $500/month worth of servers. The cost of using this much data with EC2 - around $5400 a month. It's not even close to being competitive.
I'm surprised this isn't mentioned more often.
It is hard to beat AMZN's flexibility and easy provisioning. For instance, even my sister-in-law can walk up to AWS, spin up an instance with Windows, log in with RDP and then enjoy everything you can enjoy on Windows other than high-end gaming. (For you mac-ers in the audience, this is a clean and economical way to make sure that your landing page works for "the rest of us")
But you don't have to choose because you can work with Linux. On top of that, AMZN layers services such as Elastic Map-Reduce which are compatible with industry standards but eliminate so much time you could waste sysadmining.
You actually buy provisioned IO. The throughput can be actually smaller in some scenarios than for non-provisioned IO, provided you: don't have neighbors or they are not noisy. And boy, is that provisioned IO expensive (we've bought quite a bit of EBS volumes).
It feel likes the author doesn't know that EBS is network storage and not local storage and therefore depends heavily on the size of the instance for performance (unless you buy Provisioned IOPS).
In sphinx's config files, its super easy to split an index up into 4 parts, and then assign each partial index to one core.
Doing this, we find that all 4 cores are utilized nicely (>50%) almost all of the time. I feel like we're getting a really good value out of that machine.
AirBnB has a good writeup on this config: http://nerds.airbnb.com/how-we-improved-search-performance-b...
EBS performance is an issue - Yes it is, don't use EBS. Use instance store and push to S3 for persistence. If you need performance I/O for something, there is likely a separate service available that pushes your I/O bottleneck further away (RDS, ElastiCache, SQS, etc.).
AWS costs more for weaker CPU - Indeed, this can be the case. But, it's often cheaper (but not much) to put up an Elastic Load Balancer with an Auto Scaling Group and dynamically support your peak traffic than it is to pay for an enormous VPS that sits idle 60% of the time.
As these benchmarks suggest, I agree that if you're only using one EC2 instance (and you can't get away with a micro), you should probably be investigating other solutions. If you want to architect your app/project/service/whatever to be more distributed and fault-tolerant, AWS can probably make that easier (not necessarily cheaper).
On a side note - From a marketing perspective, Amazon probably should have launched PIOPS as a separate product from EBS. The idea that "EBS sucks" is pretty firmly entrenched, so EBS+PIOPS is fighting an uphill battle.
For us it is worth the price (>$40/mo for 1 GB ram) for great stability and support. The Openstack API is really nice, too.
Way back when, Amazon gave me about $1200 of free use credits over a two year period, and I played, experimented, used it for most customer projects, etc. I also used AWS for almost all of my own projects.
In the last year or two however, I have started going back to renting large VPS by the month (I use RimuHosting, but there are a lot of good providers) because you get so much more capabilities for the same amount of money.
A little off topic, but another way I have found to save money is to wean myself off of Heroku by taking the little bit of time to set up a git commit/push hook to automatically deploy my web apps. I was using a manual deployment scheme before that took me a minute for each deployment - not so good.
All that said, AWS is really awesome for some jobs like periodically crunching data with MapReduce, etc. I bought a very useful little book "Programming Amazon EC2" a few years ago, and I recommend that as a good reference for using the AWS APIs.
I won't go into it in this comment because it's been beaten to death and you can find information about deployments everywhere; even the commenter on your blog post makes a better suggestion than your VCS deployment strategy.
Continuous integration, source/binary distributions, automated provisioning, sandboxing, versioning, etc...; it's all out there.
I would expect anyone who works with AWS to know this by now.
Secondly, if you wanted to get truly accurate results, doing this same test over an array of provisioned instances would be good, as quality can vary.
Still, interesting information nonetheless. Really surprised how much faster Linode was than DigitalOcean.
On a cost per hour basis, DO is the cheapest. Linode next. And Amazon most expensive.
Also, DO is only located in NY, while the others offer central and west cost locations.
I'm about to deploy an app and I would like to get other impressions.
I like the workflow for creating stacks and I really like the polyglot nature of selecting which cloud to deploy to.
They did have a security incident a while back and I'd recommend reading up on that. They had a blog post about it but it appears they've since pulled that. I don't know what that's about.
I'll tell you that you'll be hard-pressed to find better customer support/service in another company. They're open to suggestions and will help you if you ever run into an issue deploying an app.
http://serverbear.com/237-small-amazon-web-services#benchmar...
Well, yeah. If you use it as just commodity servers. At Radius, we migrated our index build process over to using Elastic Map Reduce on spot-priced servers and it's been a huge cost savings.
Long story short, move to Amazon if you want elasticity and can design around saving from their services. Otherwise, look elsewhere.
Indeed the CPU is limiting factor in your bench, and while you say you're comparing apples to apples, it's not true in my opinion.
Maybe I'm wrong but I never see these actors as really competing in the same market : of course if you're looking for the best CPU perfomance/price ratio AWS is one the worst choice. But that's not what you're paying for imo, as someone already mention you're paying for the programmatic access, the ecosystem, the auto-scaling ability, monitoring, etc. Move to AWS if you need the "Elastic" part in EC2 (Elastic Cloud Compute) or if you know what you're doing and what you're paying for.
Same for DO, what you're paying for is the SSD, they're advertising about it everywhere, so it's kind of obvious that a big part of what you'll pay for is the SSD.
Linode is just plain xen, and domUs packed not so tightly, so it performs more or less quickly if other instances are 100% idle.)
The main assumption about virtualization is that there is no two high load domU instances running in parallel, same scenario as it was with primitive apache virtual host based hostings - we could pack them tightly, each one have 100 requests per week on average.
So, virtualization works fine for almost always idle development servers, but everything would fall apart in a I/O intense production environment.
The mantra is "I/O request should be separated and data partitioned". In case of cheap virtualized "servers" storage is the first bottleneck, because you share it with other domUs. Once they're idle, you're OK. Should one next to you running, say, a torrent tracker - you're screwed.
It doesn't matter what you're running under - Xen or just FreeBSD's jails (still love them). The problem is that a HDD could perform only one operation (read or write) at a time.) So, your dom0 is deeply in IOwait, and your, say, mysql on domU locked all your tables, waiting for insert/update completion.
I could write a brochure, but in short - virtualization on productions is the same unnecessary complications as a Java Virtual Machine. They are nice toys, but in production everything is better without them.)
This discussion here reminds me of PC customers buying behavior in the 90ies. What's better AMD or Intel? ... Nowadays other features are key: What's the weight of this device? How thick is it even? Apple has changed the way we look at these things today.
Convenience also matters in hosting a lot. How much time do i have to spend to have my app up and running? Do i really want to set up and maintain everything myself? How good is the support? Do i want just bare metal computing resources or a solution provider with an eco system?
What matters the fastest server ever, when the queries are slow? The performance of any app/website relies heavily on the engineering skills of the developers. See caching, see i/o load, see frontend technolgies, see #perfmatters.
disclaimer: i am co-founder of a PHP PaaS.
If you need what Amazon offers, all of the AWS services or most of them, then go for it. But you don't use AWS for raw power.
This will have a significant impact on the ab numbers (particularly as he's not using the -k option and therefore establishing a new TCP session for every request).
My first impression? graphs are better :)
Well, Amazon just sucks.
