> This is the reason why we were able to easily cope with 2M+ requests in less than 24h when Midnight Society launched
This gives the answer, while it's probably not evenly distributed gives 23 req/sec (guess peak 60 - 100 might be already stretching it). Always wonder about use cases around 3 - 5k req/sec as minimum.
[edit] PS: not really ditching neither k8s pg nor AWS RDS or similar solutions. Just being curious.
That kind of load is something a decent developer laptop with an NVME drive can serve, nothing to write home about.
It is sad that the "cloud" and all these supposedly "modern" DevOps systems managed to redefine the concept of "performance" for a large chunk of the industry.
Honestly, if I had to pin it on just one thing, I'd blame networking everything. Cloud would fit as a subset of that. Networking slows things down at the best of times, and the latency distribution can be a nightmare at the worst. Few developers think about the cost of using the network, and even fewer can think about it holistically (e.g., to avoid making 50 network transactions spread throughout the system when you could do it all in one transaction if you rearranged things).
Many frameworks that make early development easy fuck you later during growth with ORM calls, tons of unnecessary text in the DB, etc.
You can get that many IOPS with aurora, but the cost is exorbitant.
I have run 500+ req/sec on a raspberry pi using 4 TB dataset with 2 GB of RAM, with under 100ms for the 99.99 percentile
A few hundreds req a second is basically nothing.
Main difference would be storage speed and how it exactly is attached to a container.
I thought this was referring to 2M+ requests per second over a ramp period of 24h, not 2M requests per 24h?
What does this syntax mean? Surely you wouldn't use big-o notation with a constant in it, especially to convey the same meaning as the thing without the O?
I could never get a straight answer on whether running a database in a container (and mounting the storage volume through a bind mount/network drive or whatever) came with a performance hit compared to running it as a systemd service for example.
TL;DR performance impact should be negligible, could be even slightly negative compared to a VM (when running K8s on bare metal).
Running HA Postgres is not easy...but at any sort of scale where this stuff matters, nothing is easy. It's not as if AWS has 100% uptime, nor is it super cheap/performant. There are tradeoffs for everyone's use-case but every thread is full of people at one end of the cloud / roll-your-own spectrum.
They seem like a huge step up from the arcane "1980s Unix" nightmare of Postgres clustering but I don't hear about them that much. Are they not used much or are their users just happy and quiet?
(These are all "NewSQL" databases.)
We run them on dedicated hosts or on Hetzner cloud instances. We tested out RDS Postgres, but that would've literally tripled our cost for worse performance.
Only had a few hiccups with the big cluster but they were resolved quickly with their support.
We're very happy with the product, and have leaned quite a few optimization tricks to get the best out of it. Easy to use as well, join the nodes and it just works.
It's not perfect though, we've had quite a few issues with deleting lots of data at once, it doesn't like that. So we have to do deleted in smaller chunks.
TiDB is a pretty interesting project, but there are a few limitations that should be taken into account when trying to use it: https://docs.pingcap.com/tidb/stable/mysql-compatibility
A lot of these are tradeoffs that will affect how a database can be architected, such as having no access to foreign keys and thus needing to think about any sort of consistency and not leaving orphaned data at the application level.
General storage IOPS is governed by your provisioned storage size. You can again get much more consistent performance by using provisioned IOPS.
Feel free to email me if you want to chat through things specific to your env - email is in my about:
Unused memory on Linux will be automatically used to cache IO operations, and you can also tweak PG itself to use more memory during queries (search for "work_mem", though there are others).
If your workload is read-heavy, just giving it more memory so that the majority of your dataset is always in the kernel IO cache will give you an immediate performance boost, without even having to tweak PG's config (though that might help even further). This won't transfer to writes - those still require an actual, uncached IO operation to complete (unless you want to put your data at risk, in which case there are parameters that can be used to override that).
For write-heavy workloads, you will need to upgrade IO; there's no way around the "provisioned IOPS" disks.
Of course if you need to recover quickly in a disaster you'll want a hot standby or replica. Still may be cheaper than PIOPs. (Especially if you need HA anyway.)
> It's often cheaper and faster to increase the instance storage than move to EBS, prioritized or not.
You're saying it may well be cheaper to increase storage in order to get more IOPS than moving to an EBS-optimized instance type?
Regarding HA, not relevant for at this point (assuming I understood you correctly). We've only got a single primary and one replica, the latter being used primarily for analytics.
Sorry for not answering everyone individually, but I see some confusion duo to the lack of context about what we do as a company.
First things first, Nhost falls into the category of backend-as-a-service. We provision and operate infrastructure at scale, and we also provide and run the necessary services for features such as user authentication and file storage, for users creating applications and businesses. A project/backend is comprised of a Postgres Database and the aforementioned services, none of it is shared. You get your own GraphQL engine, your own auth service, etc. We also provide the means to interface with the backend through our official SDKs.
Some points I see mentioned below that are worth exploring:
- One RDS instance per tenant is prohibited from a cost perspective, obviously. RDS is expensive and we have a very generous free tier.
- We run the infrastructure for thousands of projects/backends which we have absolutely no control over what they are used for. Users might be building a simple job board, or the next Facebook (please don't). This means we have no idea what the workloads and access patterns will look like.
- RDS is mature and a great product, AWS is a billion dolar company, etc - that is all true. But is it also true that we do not control if a user's project is missing an index and the fact that RDS does not provide any means to limit CPU/memory usage per database/tenant.
- We had a couple of discussions with folks at AWS and for the reasons already mentioned, there was no obvious solution to our problem. Let me reiterate this, the folks that own the service didn't have a solution to our problem given our constraints.
- Yes, this is a DIY scenario, but this is part of our core business.
I hope this clarifies some of the doubts. And I expect to have a more detailed and technical blog post about our experience soon.
By the way, we are hiring. If you think what we're doing is interesting and you have experience operating Postgres at scale, please write me an email at nuno@nhost.io. And don't forget to star us at https://github.com/nhost/nhost.
If I was launching a BaaS I wouldn't touch AWS. Grab a few Hetzner bare metal servers and setup your infra. You're leaving a massive profit margin to AWS when you don't have to.
We haven't open-sourced any of this work yet but we hope to do it soon. Join us on discord if you want to follow along (https://nhost.io/discord).
A while ago, I worked for a company that offered a hosted version of their application that required Postgres, etcd, Kubernetes, etc. It was set up so that every customer got their own GCP project, containing a K8s cluster, Cloud Storage, and a Postgres instance, The k8s cluster ("workspace") then contained dedicated nodes (4vCPU x 16G RAM at a minimum, autoscaling up according to their workload including GPU compute), SSDs, a public-facing LoadBalancer, etc. This is good for per-customer isolation, but quite costly at idle, on the order of several hundred dollars a month. Users expect this kind of isolation (but need the SOC2 and similar checkmark for sure), but they don't expect to be charged when they're not running anything, which was a problem for us.
If I was doing this again, I would do it this way, at least for the MVP. One option is to make the application multi-tenant aware, and isolate at the application level instead of at the GCP project level. This might be more difficult to get certified and might not meet everyone's HIPAA-like compliance goals, but is a good starting point, especially for free trials.
The other option that was very appealing to me is to give each user a VM that just gets de-scheduled when no requests are being made. Instead of k8s managing nodes, nodes would manage k8s. The downside there is that cluster size is limited to whatever the largest node you can buy is, but honestly, 448vCPUs is a ton (AWS's max instance size at the moment), so it's a very workable solution. When users sign up, create a VM image that runs K8s, Minio, Postgres, etc. and route traffic to it with a shared L7 router/front proxy. If their workloads autoscale up, freeze and migrate the VM to a machine with more resources. If they're not using it for a while, freeze it completely, and reprogram your front proxy to point at a program that waits for an RPC / web request and starts up the VM when one comes in. Now your idle cost is the cost of your block storage, modulo deduplication, instead of dedicated CPU cores and RAM. You also get a lot of knobs to control your actual compute cost; you aren't reliant on your users provisioning spot instances from their cloud provider, you can just tell cron jobs to run when CPU load is lowest, or set your own rate to incentive off-peak usage. And, you can pretty much get away with charging nothing for idle instances, limit free trials in aggregate to X CPU cores, etc. I think it would have been good, though complex.
TL;DR: RDS is a highly-available always-on service. But customers might not want HA or always-on. By being able to turn off the database at the right moment, you can save a lot of money on compute, which makes things like good free trials more economically viable. I think OP is on the right track to a successful k8s-based business and wish them great luck!
Sure, a decade-old dedicated team at a billion-dollar multinational corporation has honed a solution designed to support hundreds of thousands of customers with high availability, and we could pay a little bit extra money to spin up a new database per tenant that's a little bit less flexible, ..... or we could reinvent everything they do on our own software platform and expect the same results. All it'll cost us is extra expertise, extra staff, extra time, extra money, extra planning, and extra operations. But surely it will improve our product dramatically.
Not bad to invest some extra time to get better performance.
You are falling to “Appeal to antiquity” fallacy if you think something old is better.
If you're gonna spend effort in running Postgres manually, do it on bare-metal and at least get some reward out of it (performance and reduced cost).
Source: AWS user on very very large scale stuff for about 10 years now. It's not magic or perfection. It's just someone else's pile of problems that are lurking. The only consolation is they appear to try slightly harder than the datacentres that we replaced.
Probably there is also an 80/20 for most users where it's not awful if you can restore from a cold storage, say 12h, backup.
Examples of alternatives for managed Postgres:
* Supabase is $0.125 per GB
* DigitalOcean managed Postgres is ~$0.35 per GB
For small startups, my DR/HA plan is hourly delta snapshots of the whole volume.
GCP, AWS and Azure all make this possible.
EBS pricing is here: https://aws.amazon.com/ebs/pricing/
I'd have to check with the team but I'm 80% sure we're on gp3 ($0.08/GB-month).
That said, we have a very generous free tier. With AWS we have an enterprise plan + savings plan + reserved instances. Not all of these affect EBS pricing, but we end up paying a lot less than the average AWS user due to our high-usage.
So really we don’t know how much RDS was a problem compared to the the tenant distribution.
For the purposes of an article like this it would be nice if the two steps were separate or they had synthetic benchmarks of the various options.
But I understand why they just moved forward. They said they consulted experts, it would also be nice to discuss some of what they looked or asked about.
Sure, RDS is expensive, but it's also quite well done. Almost every cloud platform service is more expensive than doing it yourself. No surprise here.
In the past I've deployed SQLite over Postgres for cost cutting reasons. It's not too difficult to swap out unless you're heavily bought into database features.
In a business environment, this is actually not true unless you consider the extreme long term.
A Multi-AZ MySQL RDS instance of size db.m1.large (2x vCPU, 7.5GB of RAM), a 500GB standard disk, and an on-demand pricing model with 100% monthly utilization, will cost you approx. US$7,000 per year (rounding up.) That price gets you almost everything you can imagine from that service.
US$7,000 wouldn't get you my services for the time needed to setup a service that came even 30% as close in terms of reliability, feature parity and support.
RDS is not expensive (in the right environment.)
For a small startup or operation, a managed service having credible snapshots, PITR backups, failover, etc. is going to save a business a lot of ops cost, compared to DIY designing, implementing, testing, and drilling, to the same level of credibility.
One recent early startup, I looked at the amount of work for me or a contractor/consultant/hire to upgrade our Postgres recovery capability (including testing and drills) with confidence. I soon decided to move from self-hosted Postgres to RDS Postgres.
RDS was a significant chunk of our modest AWS bill (otherwise, almost entirely plain EC2, S3, and traffic), but easy to justify to the founders, just by mentioning the costs it saved us for business existential protection we needed.
We're offering free projects (Postgres, GraphQL (Hasura), Auth, Storage, Serverless Functions) so we need to optimize costs internally.
Do I have to manually upgrade my old instances?
We're working on a one-click migration from RDS to a dedicated Postgres instance for older projects. Should be live in the next week or so.
but. It sounds bloody expensive to develop and maintain a reliable psql service on k8s
Network IOPs and NAT nastiness or disk IO the bigger issue ?
