Framework Framework Index
Gemini 87.88
Vert.x 76.29
Express 68.85
Sinatra-Ruby 67.88
WebGo 51.08
Compojure 45.69
Rails-Ruby 31.75
Wicket 29.33
Rails-Jruby 20.09
Play 18.02
Sinatra-Jruby 15.96
Tapestry 13.57
Spring 13.48
Grails 7.11
Cake 1.17
I too wanted Play to show higher numbers. There's certainly a possibility we have something configured incorrectly, so we'd love to get feedback from a Play expert.
About the cost per additional line of code for PHP, it mainly comes from not having an opcode cache and having to load and interpret files on every visit. mod_php was and will always be trash. I commented earlier about it too.
In case of Ruby, and talking about Rails, even when using Passenger, the rails app is cached via a spawn server. That's not the case with PHP.
Similarly, Python has opcode cache built-in (.pyc files). Also, I am not sure about gunicorn but others do keep app files in-memory.
Servlet does the same about keeps apps in memory. You get the idea.
Frameworks definitely have an impact but it's very hard for one person to know the right configuration for every language. You had done some good work there, but it will take a lot of contributions before the benchmark suite becomes fair for every language/framework.
We're hoping to post a follow up in about a week with revised numbers based on the pull requests and any other suggested tweaks we have time to factor in. We're eager to see the outcome of the various changes.
See e.g. https://github.com/playframework/Play20/blob/master/document... and https://gist.github.com/guillaumebort/2973705 for more info.
Gemini is sort of an outlier that doesn't really fit either category well, but the micro-frameworks have a fairly consistently higher framework optimization index than the large MVC frameworks which is as expected.
Express and Sinatra really stand out as widely-used, very high percentage of platform performance retained frameworks here. I've never used Vert.x, but I will certainly look into it after seeing this. I'm very impressed that Express is so high on this list when it is relatively young compared to some of the others and the Node platform is also relatively young.
Play seems particularly disappointing here since it seems any good performance there is almost entirely attributable to the fast JVM it's running on. Compojure is also a bit disappointing here (I use it quite a bit).
cake 18.9% (312 vs 59)
compojure 12.1% (108588 vs 13135)
django 16.8% (6879 vs 1156)
express 16.9% (42867 vs 7258)
gemini 12.5% (202727 vs 25264)
go 13.3% (100948 vs 13472)
grails 7.1% (28995 vs 2045)
netty 18% (203970 vs 36717)
nodejs 15.6% (67491 vs 10541)
php 11.6% (43397 vs 5054)
play 20.6% (25164 vs 5181)
rack-jruby 15.6% (27874 vs 4336)
rack-ruby 22.7% (9513 vs 2164)
rails-jruby 22.7% (3841 vs 871)
rails-ruby 20.7% (3324 vs 687)
servlet 13.4% (213322 vs 28745)
sinatra-jruby 21.2% (3261 vs 692)
sinatra-ruby 22.2% (6619 vs 1469)
spring 7.1% (54679 vs 3874)
tapestry 5.2% (75002 vs 3901)
vertx 22.3% (125711 vs 28012)
webgo 13.5% (51091 vs 6881)
wicket 12.7% (66441 vs 8431)
wsgi 14.8% (21139 vs 3138)
I found it interesting that something like tapestry took a 20x slowdown when going from dedicated to ec2, while others only took ~5x slowdown.
Edit: To hopefully make it clearer what the percentages mean - if a framework is listed at 20%, this means that the framework served 1 request on ec2 for every 5 requests on dedicated hardware. 10% = 1 for every 10, and so on. So, higher percentage means a lower hit when going to ec2.
Disclosure: I am a colleague of the author of the article.
So, in responses per second, the throughput on ec2 was 5.2% that of dedicated hardware, or approximately 20 times less throughput. The use of the word slowdown was possibly a bad choice, as none of my response had to do with the actual latency or roundtrip time of any request.
