Also don't forget that you are load testing all the dependencies of your service. Database, caching tier, external services, etc. Make sure other teams are aware!
Also nothing beats real world traffic. Users' connections will stay open longer than a synthetic tool may hold them open due to bandwidth, they make very random, sporadic requests too. Your service will behave very differently under large amounts of real world traffic vs synthetic.
Other options if you are running multiple web servers is to shift traffic around to increase traffic to 1 host and see where it fails. That is usually a very reliable signal for peak load.
And don't forget to do this on a schedule as your codebase (and your dependencies codebases) changes!
Guilty. I've had one of our partners call me one time because I'd caused a huge load on their end. Lots of apologies and embarrassment followed.
Later I mocked out those external calls with stubs which behaved similarly, in that I could specify the min/max/average wait times and error rates.
wrk does this with lua. https://github.com/wg/wrk/blob/master/src/wrk.lua
Also even things like the venerable jmeter supported pulling parameters from a csv file.
Once you've got that out of the way, don't forget that you'll want a distribution story. It does not matter how efficient your tool might be on a single machine - you'll want to distribute your tests across multiple clients for real-world testing.
"Sure it's easy" you might say, "I know UNIX. Give me pssh and a few VMs on EC2". Well, now you've got 2 problems: aggregating metrics from multiple hosts and merging them accurately (especially those pesky percentiles. Your tool IS reporting percentiles rather than averages already, right?!), and a developer experience problem - no one wants to wrangle infra just to run a load test, how are you going to make it easier?
And, this developer experience problem is much bigger than just sorting out infra... you'll probably want to send the metrics produced by your tool to external observability systems. So now you've got some plugins to write (along with a plugin API). The list goes on.
I'm very biased, but it's 2024. Don't waste your time and just use https://www.artillery.io/
1. https://www.artillery.io/blog/load-testing-workload-models
Shameless self promotion but I wrote up a bunch of these issues in a post describing all the mistakes I have made so you can learn from them: https://shane.ai/posts/load-testing-tips/
If your desired load for testing is small, it's not a big deal, of course.
Ruby with browser:
https://browserup.com/docs/en/load/ruby-load-test.html
Command-line installation:
We're currently trying to have each rails model implement a #new_example method that builds a valid subgraph filled in by Faker, ready to save. Ie a
user = User.new_example
Still early, we'll see how it goes.
We generate data based off of your database schema and your production data (if you give us access.)
Since you've kinda already built something like this I would be curious to hear what you think!
Talking about efficiency being a priority, but using RoR. I guess that is one way of saturating the CPU.
Most bottlenecks are either that database choices or poor code/design choices by developers. That is especially true today.
A coworker made similar claims to me about Laravel, but the framework really encourages you to do half a dozen database queries in even a pretty minimal request, and for example implemented bulk inserts as a for loop that did single inserts. If you didn't know better with an access pattern like that, you might think the database is the bottleneck long before it actually should be. Is Rails different? My sense was they are very similar.
That said, in my experience, CPU is often the ultimate bottleneck with PHP, Ruby, Python, and.. Like everything. Over the years serializers have often been a pain point; XML in PHP and RoR, and the Rails "serializers" currently. Any sort of mapping or hydration(which is a LOT of what happens in web apps) is comparatively slow, often order of magnitude or more, over something like nodejs, C#, golang, and etc.
> Most bottlenecks are either that database choices or poor code/design choices by developers
Perhaps in sheer quantity, but with experience those are often low hanging fruit. After those are addressed you are left with the pain of the language and framework inefficiencies.
Rails has really poor startup time due to loading all codepaths. We switched to Django and it runs beautifuly on AWS Lambda where our CI is more expensive than actual server costs. We're a b2b application so traffic is quite low so we REALLY don't saturate the CPU in a normal Fargate setup.
Our ~500k lines app takes multiple seconds to start, which is why I'm not really investigating a lambda-style setup... Do you have specific strategies to make startup fast?
Definitely times it isn't true. But if you're not doing a load test bc it's a pita, do it locally. Most of the time I've wanted to do this, all the action is inside the app. Just be careful to acknowledge that there could be limitations / surprises.