I can think of several metrics that would come first - even for sites with fairly high performance requirements.
How often is the web framework genuinely the bottleneck (at least in a way that's not easily fixable with caching or any other similar sledgehammer)?
Personally, it helped me realize that my mistaken belief that nodejs was hella fast and that old Java based frameworks are slow. It also introduced me to Kotlin (I started on it to try Kooby) and overall love that I can at a glance see that for my chosen set of languages, which farmeworks-dbs are a good bet.
Which surely are common but long(er) running database requests or delegation to another service aren't super rare either.
Similar some of the frameworks might have major reliability problems which can randomly "crop up" due to thread pinning and similar.
I believe it should have some form of "chaos" test where instead of interacting with the DB it interacts with a another test server on both ends and the front and back connected server run through long random "chaos" simulation (and another version where a pre-recorded run is used for reproducibility).
While the benchmark focus on more req/s, we can also see it the other way around: for the same req/s, you can use smaller hardware, i.e. save money and energy.
It has lead me to treat these benchmarks with a grain of salt.
There are very few circumstances that I can imagine that would send me running for C++ for a web project. I'm much more concerned about balancing efficiency of the developer environment with performance, security and maintainability, than pure performance. At the point that I'm not, my priorities are out of whack (aside from very niche situations). The side effect of these benchmarks, while we all love benchmarks, is the idea that any of these faster frameworks are actually a better choice because of a benchmark result.
Unless it is super tuned, running on a specialise VM, I seriously have doubt about that. It is already pretty damn good if PHP/Python Django and Ruby Rails could do 10ms request excluding the Database on a moderately large App. Generally they are 20ms+.
And recently shopify even have their performance matrix allowing for 300ms per request.
A web search for "lithium web framework" gave https://li3.me as the first result, which is a PHP framework.
Why did you pick such a name if already taken by a similar product?
Is it the use of some Linux only kernel API that enables it to be slightly faster than the other frameworks?
Why did you create lithium?
To see how much C++ can get closer to dynamic langagues typically used for web programming and at the same time make absolutely no compromise on performances.
How long have you been working on it?
8 months on lithium, but I already wrote another framework (silicon, which I rewrote from scratch because of slow compilation time).
What do you do professionally?
I worked for several company, doing different things including, real time image processing, neural nets for photography, web programming, robotics
What are your plans for it?
No big plans yet, just having fun for now, but I'd like to keep implementing the features if the users ask for it, improve the doc. In the more long term, building websockets would be nice too, and other services on top of it.
I thought there were no zero cost abstractions in C++?
Still, it isn’t RoR or anything JS so it has that going for it.
Edit: Comment lower down suggests: https://github.com/matt-42/lithium is the correct one.
"Lies, Damn Lies, and Benchmarks" is somewhat true in this like any benchmark, but as others have pointed out, important techniques gain exposure using this benchmark over the years that trickle into mainstream use.
It would be nice if a summary of what techniques were used by new "frameworks" to gain leadership. All too often discussions of this benchmark turn into pissing matches by language zealots.
Actix's turn at the top was very interesting for the, uh, discussion, it prompted in the rust community about unsafe, although it was stressful for the original author.
In javaland, Vert.X has been high on the list for quite a while, but as I understand it, it uses "arena" allocation of buffers to avoid GC and other tricks to get the JVM up to par with non-GC languages, but isn't really typical of JVM development.
I would also like to know if HTTP protocol "hacks" are used, or networking tricks like userland network stacks.
But alas, it's just a huge wall of github projects and not a lot of use to a run of the mill developer like me.
Seriously why is this reaching the front page, who really care about such benchmark ?
Some of the ideas developed in these frameworks percolate down to the PLs that use them and to the libraries they use, which benefits everyone. These frameworks also test new language features.
ASP.NET Core is the newer (complete rewrite, inc runtime), leaner cross-platform (Win,Linux,macOS) Web Framework that's built on top of the performance focused .NET Core runtime released in 2016 https://dotnet.microsoft.com/learn/aspnet/what-is-aspnet-cor... They're only the same by name which is unfortunate, because it's completely new. It's the future runtime/platform for .NET and will be renamed to .NET 5 whilst the older .NET Framework will stay at v4.x (currently at v4.8).
I certainly wouldn't say that. Full Framework .Net and .Net Core are _far_ more similar than they are different - they share a very large common base class library API surface in Net Standard.
A huge percentage of .Net MVC 3-5 (full framework) applications could be migrated to .Net Core MVC in a matter of hours to days.
You don't even have to specifically target .Net Core in a lot of cases - v2 included a compatibility shim so that libraries targeting full framework just work as long as they don't call any of the API's that were in full framework but are not in the Core runtime.
The main differences are in startup configuration of your web app and how you go about implementing cross-cutting concerns that might inspect or intercept every request - but even for a lot of common things like authentication there are similar extension points to what existed previously.
There are some particular technologies - like WCF Server or Linq2Sql - that are hard blockers if you were relying on them - but by and large moving from older full framework .Net to .Net Core is not that difficult and certainly doesn't require learning much new, especially fundamentals - it's mostly just some details of the web app framework.
The industry grade server ecosystem that is the JVM use the jdbc which sadly has a blocking socket thus not allowing asynchronicty. But when loom arrive every jdbc existing code will magically, automatically become truly asynchronous such spring should come on the top 4 place.
There is also a wrapper of the jdbc through kotlin coroutines and there are reactive jdbc implementations such as R2DBC. It is unclear as of today if such solution enable postrgresql async queries and batch processing. It seems that nobody has tried those on TechEmpowerUp which is sad. Finally one could use libpq over JNI. Edit: I have read that the next release of pgjdbc (43) will switch from std socket to the NIO non blocking socket.
What should be heuristically the fastest HTTP framework (H2O, in C) has refused to use the old libpq fork because the api is not stable and thus not production grade.
FYI lithium is not using a libpq fork from 2016, it has been rebased on master 1 month ago. but yes Lithium (the sql-pipeline branch) is using it. (I mailed the drogon maintainer to update it aswell).
About H2O, the non batched version of lithium is as fast (slighly faster on some tests, slighly slower on others), while being much simpler to use (implementation of TFB of H20 is 4400 lines vs 250 lines for lithium...)
What would to you the other major optimizations?
I know less than zero about Java and Loom, but if I understand correctly loom enables first class continuation and seamless M:N threading.
But I do not see how that will help jdbc, (unless you have tens of thousands of db sessions); I understand that the power of the new libpq is the ability to pipeline queries and that won't magically appear without development effort.
Assuming connection pooling, can't you just use another Thread to do your queries?
[0] https://github.com/TechEmpower/FrameworkBenchmarks/tree/mast...
Optimize your webapp before hunting for the fastest frameworks.
In web applications, apart from glaring mistakes such as N+1 queries, pretty much every bottleneck I had in production was due to slow serialisation, slow database abstraction or excessive CPU/memory usage due to language/framework. Also had issues caused by initialisation time. Saving 200ms or 300ms on each request can give you better user experience.
For growing startups, having a fast web framework gives you the choice to postpone doing complex horizontal-scaling for months, or even years. This saves money on the sort term and allows you to keep growing their products without having to worry about other things. The database can scale vertically in the meantime.
As for large tech companies: at my previous workplace there were two migrations in the span of two years because of performance issues. Hundreds of developers collectively agreed on changing the tech twice because of framework bottlenecks.
Of course you shouldn't go straight into a C++ framework when Rails becomes too slow for you, but people should know the tradeoffs of the framework they're using.
Believe me (or not) when I say that Amazon, among all FAANGs, could be the most data-driven, at least in the teams I knew, almost to an excess.
We execute runs like this continuously to allow maintainers of test implementations to observe the results of their contributions. Given the performance seen here, it is very likely that Lithium will be well ranked in the next official round. But we feel the ranking seen in a continuous run such as this should be taken with a grain of salt until that next official round is available.
[1] https://tfb-status.techempower.com/results/57b25c85-082a-401...
"Only cases where the structure of the object is known at compile time are covered."
Numbers are impressive, but there are some big trade-offs that come with.
My personal criticism is that C++ is usually (not always, though) the wrong language for implementing web-facing applications or APIs. There is a considerable advantage in terms of speed (when using the right framework) but there are so many things that can subtly go wrong in terms of safety and security that you need to be very, very careful when picking C++.
And I write this as a C++ developer.
"Rapid Web Application Server (in Assembler)":
Discussed on hn a few times as I recall:
https://news.ycombinator.com/item?id=9948749
(see also sshtalk):
Only the profiler will tell you if there is a hidden malloc that is slowing down all the rest, or a inefficient datastructure, or ...
There is probably lot of non optimized code in lithium, it just has no impact (or if it has I still has to find it).
That said, for other kind of software (usually more CPU bound), going down to assembly or using SIMD can be the only way to get good perfs.
I always check any lesser known framework to see which ballpark it falls into. I'm always surprised to see that so many of the popular frameworks are ~10x worse than the best--although many of those 'best' don't do as much processing. I'm much more likely to pay attention to the error counts, max latency, or SD (σ).
I just took some small peak at atix (because I happen to know it) and while first all looked fine it wasn't quite that realistic.
Mainly:
- It uses a fork of tokio-postgres specific for the test (which differs in that Client is no longer send and it has a Unsafely soundness hole by wrongly using Unsafe Cell, and no Issue tracking enabled on, through replacing that cell with RefCell probably yield very similar performance)
- Instead of using the default web::Json responder it uses simd-json (EDIT: explicitly encoding the data into a buffer instead of returning it wrapped in web::Json)
- Uses snmalloc instead of the default allocator (https://github.com/microsoft/snmalloc)
Just to be clear besides the first point all of this is not unrealistic for a context where you want to highly optimize your server. But not how actix is used most of the time.
---
Edit: Also just to be clear I didn't nit pick intentional on actix, it just happen to be the framework I'm more familiar with and especially using snmalloc for actix seems to be quite a reasonable idea.
EDIT2: The Unsefety unsoundness isn't triggered given the way this library is used, it's still there and would prevent this change from ever been merged upstream.
not obvious from a google search
Rust, C++ just said “hi” to you again, it’s on. Time to make Actix fast.
