Without any intent to insult what you've done (because the information is interesting and the writeup is well done)... how do the numbers work out when you account for actually implementing and maintaining the database?
- Developer(s) time to initially implement it
- PjM/PM time to organize initial build
- Developer(s) time for maintenance (fix bugs and enhancement requirements)
- PjM/PM time to organize maintenance
The cost of someone to maintain the actual "service" (independent of the development of it) is, I assume, either similar or lower, so there's probably a win there. I'm assuming you have someone on board that was on charge of making sure Aurora was configured / being used correctly; and it would be just as easier if not easier to do the same for your custom database.
The cost of 120,000/year for Aurora seems like it would be less than the cost of development/organization time for the custom database.
Note: It's clear you have other reasons for needing your custom database. I get that. I was just curious about the costs.
This sounds what big companies or a disorganized company would need. For an efficient enough company, a project like this needs just one or two dedicated engineers.
In fact, I can't imagine why this project needs a PM at all. The database is used by engineers and is built by engineers. Engineers should be their own PMs. It's like we need a PM for a programming language, but no, the compiler writer must be the language designer and must use the the language. Those who do not use a product or do not have in-depth knowledge in the the domain should not be the PM of the product.
Maybe for a research project or a hobby project, but not for a real, high performance database to be used in a business-critical application.
FTA:
"Databases are a nightmare to write, from Atomicity, Consistency, Isolation, and Durability (ACID) requirements to sharding to fault recovery to administration - everything is hard beyond belief."
>Engineers should be their own PMs.
For small projects, sure (your "one or two dedicated engineers"). But once you start tackling projects that require larger teams, or even teams of teams, you need someone to track and prioritize the work remaining and the work in progress (as well as the corresponding budgets for personnel, services, and other resources). Similar to the way a sole proprietor can do their own accounting, but a multi-million dollar business probably should have an accountant.
As an aside, I wonder if this might be a use case for a bitmap db engine like Featurebase (https://www.featurebase.com/).
The trick is that they didn't need a database that provides "Atomicity, Consistency, Isolation, and Durability (ACID)". By only implementing what they need they were able to keep the project small.
It's like people are scared of doing anything without making it into some huge multi hundred developer effort. They've written a super simple append only document store. It's not rocket science. It's not a general purpose arbitrary SQL database.
So that is at least 20k a month, for fairly cheap engineers.
What about when two different projects have two different requirements they need supported by the database. Which one is implemented first? What about if there is only engineering capacity to implement one?
I don’t think a database is the place for “just send a PR for adding your required feature and ping the team that owns it” kind of development. It requires research, planning, architecture review, testing, etc. It’s not a hobby project, it’s a critical tool for the business.
One of them. This is true whether you have a person named "PM" or not. It's just a matter of who picks.
> What about if there is only engineering capacity to implement one?
How does naming some guy "PM" solve the issue? The team just picks one of the features.
Only if they planned on hiring someone just to develop this new database and if they switch to Aurora they’d let them go immediately. If the said developer was already costing them $250k to maintain and develop the application and work on top of Aurora cost seems like a good way to save $100k/year.
But at the same time, Aurora costs could also scale with usage. It may cost $120k one year, $180k next year, $500k the year after. If the database they have now is well designed after it's already built it may not need active development every year but adding a feature here and there. Also, switching back to Aurora could also be an opportunity cost "we should have written our own thing and could have saved millions ...".
$60,000 per year in-house vs $1,200,000 per year aurora. No brainer really.
Note 'instances' eg plural, versus a singular EBS. There is some ambiguity here, I'm not sure where the 10x came from, but it seems plausible.
They're operationally using a funny spelling of SQLite and I don't imagine anyone arguing that such a thing needs constant attention.
Also they said their current volume is around 13k/second. They’ve built the new platform for 30k/sec per node. This should last them a long time with minimal maintenance.
They may or may not be doing other things depending on the company size and state.
You could drop the PM, engineers writing for engineers don’t need a PM.
You will likely hire similarly costed engineers to maintain the database stack anyways.
You basically hit all the talking points big cloud has brainwashed people into thinking into being true, but every day we see stories of a handful of engineers doing something we are told can’t be done and saving millions in cloud cost.
It’s so painful to watch. Software Engineering became a thing because you could hire a engineer solve your problem, and big businesses stepped in and told you that your problem was something else and gone out of its way to stifle innovation by settling industry standards on how to do things that only guarantee you use cloud services.
Any company that wants to own its destiny knows to stay away from lock-in.
They mention 13.5k simultaneous connections. The US has 4.2 million tractors alone, just the US, just tractors. If they get 10% of those tractors on the network that's a 30x to their data storage needs. So multiply that across the entire planet, and all the use cases they hope to serve.
Investing time early on so that they can store 50x data-per-dollar is almost certainly time well spent.
S3 is relatively cheap.
If this db requires 1 full-time developer then the cost would immediately be not worth it (assuming salary + benefits > $120k/yr)
As you say, without details it’s hard to know if this was a good idea.
A large part of the value of popular platforms is precisely that they are not bespoke. You can hire engineers with MySQL/Postgres experience. You cannot hire engineers who already have experience with your bespoke systems.
That barely qualifies for the median mortgage in the US.
What do you need them for?
I also wrote a KV system to keep track of metadata (tags) for an object store I invented. I discovered that it could also be used to create relational tables and perform fast queries against them without needing separate indexes.
I started calling it a database and many people complained that I was misusing the term because it can't yet do everything that Postgres, MySQL, or SQLite can do.
Databases have a long history that reaches back much further than the modern, full featured SQL databases we have today. What you built sounds like it would fit in well amongst the non-sql databases of the world, like Berkeleydb, indexeddb, mongo, redis, and so on.
Git is also a database. I got into this argument with someone when I proposed using Github as a database to store configuration entries. Our requirements included needing the ability to review changes before they went live, and the ability to easily undo changes to the config. If your requirements for a DB include those two things, Github is a damn good database platform! (Azure even has built in support for storing configurations in Github!)
There are more types of databases than those that end in "SQL".
A CSV file alone is a database. The rows are, well, rows. So is a DBM file, which is what MySQL was originally built on (might still be). Or an SQLite file.
The client or server API doesn't have to be part of the database itself.
This sounds like a database to me.
Sure, because it is common for people to mix a "database" (aka: data in some kind of structure) with a paradigm (relational, SQL, document, kv) with a "database system" aka: and app that manages the database.
Get that engineer a sales gig, that's insane upselling of the reality: git commit -am 'added array to store structs'
It also seems that just about every open source "datadog / new relic replacement" is built on top of ClickHouse, and even they themselves allege multi-petabyte capabilities <https://news.ycombinator.com/item?id=39905443>
OT1H, I saw the "we did research" part of the post, and I for sure have no horse in your race of NIH, but "we write to EBS, what's the worst that can happen" strikes me as ... be sure you're comfortable with the tradeoffs you've made in order to get a catchy blog post title
InfluxDB, the most popular time-series database, is optimised for a very specific kind of workloads: many sensors publishing frequently to a single node, and frequent queries that are not going far back in time. It's great for that. But it doesn't support doing slightly advanced queries such an average over two sensors. It also doesn't scale and is pretty slow to query far back in time due to its architecture.
TimeScaleDB is a bit more advanced, because it's built on top of PostGreSQL, but it's not very fast. It's better than vanilla PostGreSQL for time-series.
The TSM Bench paper has interesting figures, but in short ClickHouse wins and manage well in almost all benchmarks.
https://dl.acm.org/doi/abs/10.14778/3611479.3611532
Unfortunately, the paper didn't benchmark DuckDB, Apache IoTDB, and VictoriaMetrics. They also didn't benchmark proprietary databases such as Vertica or BigQuery.
If you deal with time-series data, ClickHouse is likely going to perform very well.
In what way?
I'm guessing it doesn't matter for their use-case which is a good thing. When you realize you only need like this teeny subset of db features and none of the hard parts writing you own starts to get feasible.
Where C* databases seems to fall down are point updates and in this case, requirement to implement your own aggregations.
For these workloads you are much better off (unless you are already running C* somewhere and are super familiar with it) with something like Clickhouse or if you need good slice and dice then Druid or Pinot.
Sounds like a great place to work.
> There's just too much can go badly wrong, for all the sunk cost in getting anything up and running.
Engineering is the art of compromise. In many cases the compromises would not be worth it, but that doesn't mean there are zero places where it would be, and eschewing the discussion out fear of how it would be perceived is the opposite of Engineering.
1. Write your own database
2. ???
3. Profit!
Atomic: not applicable, as there are no transactions. Consistent: no, as there is no protection about losing the tail end of writes (consider "no space left on device" halfway through a record). Independent: not applicable, as there are no transactions. Durable: no, the data is buffered in memory before being written to the network (EBS is the network, not a disk).
So with all of this in mind, the engineering cost is not going to be higher than $10,000 a month. It's a print statement.
If it sounds like I'm being negative, I'm not. Log files are one of my favorite types of time series data storage. A for loop that reads every record is one of my favorite query plans. But this is not what things like Postgres or Aurora aim to do, they aim for things like "we need to edit past data several times per second and derive some of those edits from data that is also being edited". Now you have some complexity and a big old binary log file and some for loops isn't really going to get you there. But if you don't need those things, then you don't need those things, and you don't need to pay for them.
The question you always have to ask, though, is have you reasoned about the business impacts of losing data through unhandled transactional conflicts? "read committed" or "non-durable writes" are often big customer service problems. "You deducted this bill payment twice, and now I can't pay the rent!" Does it matter to your end users? If not, you can save a lot of time and money. If it does, well, then the best-effort log file probably isn't going to be good for business.
Unless your company is so far past product market fit that it hires qualified applicants by the classfull or whatever-it-is is their product, they have no business coding up custom infra bits. The opportunity cost alone is sufficient argument against, though far from the only one.
My guess for "why didn't they use something off the shelf" is that no existing software would be satisfied with the tradeoffs they made here. Nobody else wants this.
If those were your requirements, why on earth are you using Aurora?
Aurora is a multi-region, failover protected, backup managed service.
This isn't. It would have been cheaper and quicker to install an OpenSource logging DB on an EC2. Like Elastic.
Aurora is a nice drop-in replacement for Postgres. Kind of a good way of dealing "oh shit, I built way too much stuff on top of a database that can't scale to this workload." It was good enough to find product/market fit, but now it's too slow. Solution: pay Amazon some money. If that works, that works.
If it doesn't, then you need to take a different approach. At this point, you're like 8 levels of weirdness down the stack, and that's when it's time for innovation. Nobody says "I need to make a CRUD app, so I'm going to build a team to build a database first." No! You just use Postgres!
When that starts failing in a measurable business metric kind of way, then your creative juices start flowing and you start thinking about making your own database.
(An aside: I make a database at work. We use Postgres for all the stuff that doesn't need to scale. "Is this auth token one associated with an authorized user?" That's Postgres. "Are these bytes being written to object storage approximately the same as bytes that have been written before?" That's our thing.)
(Another aside: My all-time favorite database is Spanner. I used that a lot when I worked at Google. I remember someone on my team working on the Spanner integration with our product, which was basically an in-memory database, but we needed durability. I've never seen a piece of software laugh at our 20,000 writes per second workload so hard. I imagined outages, rollbacks, furrowed brows... nope. I thought there was no way a durable replicated storage system could handle our workload. I was wrong. It just accepted all of our data and never caused a problem. So to me, that was the solution to every problem. But after I left Google, my thought was to use Cloud Spanner for everything. I quickly realized I did not have enough money to pay for Spanner! Nobody had enough money. So postgres is what I settled on. I have enough money to afford it ($0 is the starting cost) and it has never done me wrong. So I don't blame OP for starting there. It's what I would have done, anyway. BTW, I think they lowered the price a lot since I last looked, so if it sounds like the kind of thing you need, you should probably just buy it. But never underestimate "beefy computer with Postgres". By the time that fails you, you will probably have a lot of money if people like the thing you're making.)
What they say is that the logic is embedded into their server binary and they write to a local EBS. But what happens when they have two servers? EBS can't be rw mounted in multiple places.
Won't adding the second and more servers cause trouble like migrating data when new server joins the cluster, or a server leaves the cluster?
I understand Aurora was too expensive for them. But I think it is important to note their whole setup is not HA at all (which may be fine, but the header could be misleading).
> Amazon EBS offers a higher durability volume (io2 Block Express), that is designed to provide 99.999% durability with an annual failure rate (AFR) of 0.001%, where failure refers to a complete or partial loss of the volume.
if they take snapshots often enough to feel comfortable with that low failure rate, it does seem kind of reasonable to me. really low risk of a given volume failing.
you could spinup new EBS from the backup when the first region fails or keep a warm copy there, but seems like a lot of extra engi work.
> This storage engine is part of our server binary, so the cost for running it hasn’t changed. What has changed though, is that we’ve replaced our $10k/month Aurora instances with a $200/month Elastic Block Storage (EBS) volume. We are using Provisioned IOPS SSD (io2) with 3000 IOPS and are batching updates to one write per second per node and realm.
I would be curious to hear what that "1 write per second" looks like in terms of throughput/size?
They also said 30GB per month which works out to 0.7MB/sec if load is perfectly constant.
How does 0.7MB/sec end up costing $10k/mo in a hosted database?
Can you not achieve 1MB/sec of "queued writes" or something clever against SQLite?
Even moderately sized Kafka clusters can handle the throughput requirement. Can even optimize for performance over durability.
Some limited query capability with components such as ksqldb.
Maybe offload historical data to blob storage.
Then again, Kafka is kind of complicated to run at these scales. Very easy to fuck up.
Operationally there are some annoying things in OSS Kafka (hot partitions, controller failover slowness pre-KRaft, etc) but it's overall bog simple and easy to work with if you can accept the things it doesn't do (queue-like behavior).
I don't love Kafka these days but the fear mongering is a bit much.
Sidenote: If you think you want Kafka you should probably check Pulsar first, in most cases you probably want Pulsar or due to changing requirements you would have been better off going Pulsar from the start.
Stream the events to s3 stored as Parquet or Avro files, maybe in Iceberg format.
And then use Trino/Athena to do the long term heavy lifting. Or for on-demand use cases.
Then only push what you actually need live to a Aurora.
What would you use for streaming directly to s3 in high volumes ?
But somehow they are ingesting the data over network. Would writing files to s3 be slower than that? Otherwise you don't need much more than a RAM buffer?
Edit: to be clear, kafka is probably the right choice here, it is just that kafka and me is not a love story.
But it should be cheaper to store long term data in s3 than storing it in kafka, right?
As a demo, I've recently implemented a tool to browse 50 billion airplane locations: https://adsb.exposed/
Disclaimer: I'm the author of ClickHouse.
The project I believe still appears in success story on JGroups website after 20+ years. I am surprised people are writing their own databases for location storage in 2024 :). There was no need to invent new technology in 2002 and definitely not in 2024.
Sure we have had roads for hundreds of years but they're not the same ones we have today, even though it's the same concept and function.
You can't just take advantage of the technology we have today and at the same time refuse to acknowledge it and hand-wavingly claim it was just the same in '02. It's easy to say so about anything if we're going to conveniently gloss over the details.
Believe it or not, many technologies that changed the world and we depend on today actually have origin stories similar to the one shared in the article. Many of them started as custom internal tools that I imagine you would've been similarly critical towards them for trying to invent new technology needlessly.
Not a negative though, not everything needs a general purpose database. Clearly this satisfies their requirements, which is the most important thing.
https://en.wikipedia.org/wiki/Database#Database_management_s...
I'm sure they learned a lot, but probably a waste in the long run
I think everything is cheaper than cloud if you do it yourself when you don't count staffing cost.
> Databases are a nightmare to write, from Atomicity, Consistency, Isolation, and Durability (ACID) requirements to sharding to fault recovery to administration - everything is hard beyond belief.
Then talk about their geospatial requirements, PostGIS etc, making it seems they need geospatial features ("PostGIS for geospatial data storage" -- wtf? you need PostGIS for geospatial query not merely storage...)
In reality, they did not require any of the features they mention throughout the article. What a weird write-up!
I guess the conclusion is "read the F*-ing specs". Don't grab a geospatial DBMS just because you heard the words "longitude" and "database" once.
…that’s not something to brag about.
> EBS has automated backups and recovery built in and high uptime guarantees, so we don’t feel that we’ve missed out on any of the reliability guarantees that Aurora offered.
It may not matter for their use case, but I don't believe this is accurate in a general sense. EBS volumes are local to an availability zone while Aurora's storage is replicated across a quorum of AZs [0]. If a region loses an AZ, the database instance can be failed over to a healthy one with little downtime. This has only happened to me a couple times over the past three years, but it was pretty seamless and things were back on track pretty fast.
I didn't see anything in the article about addressing availability if there is an AZ outage. It may simply not matter or maybe they have solved for it. Could be a good topic for a follow up article.
[0] https://aws.amazon.com/blogs/database/introducing-the-aurora...
> [We need to cater for] Delivery companies that want to be able to replay the exact seconds leading up to an accident.
> We are ok with losing some data. We buffer about 1 second worth of updates before we write to disk
Impressive engineering effort on it's own though!
I especially like Clickhouse, it's generic but also a powerhouse that handles most things you throw at it, handles huge write volumes (with sufficient batching), supports horizontal scaling, and offloading long-term storage to S3 for much smaller disk requirements. The geo features in clickhouse are pretty basic, but it does have some builtin geo datatypes and functions for eg calculating the distance.
Sure it won't cover the bazillion cases the DBs out there do but that's not what you need. The source code is small enough for any team member to jump in and debug while pushing performance in any direction you want.
Cudos!
The one thing they do say is "no ACID". That implies no b-trees, because an unexpected stop means a corrupted b-tree. Perhaps they use a hash instead, but it would have to be a damned clever hash tree implementation to avoid the same problem. Or perhaps they just rebuild the index after a crash.
Even a append only log file has to be handled carefully without ACID. An uncontrolled shutdown in more file systems will at leave blocks of nulls in the file and 1/2 written blocks if they cross disk block boundaries.
It's a tantalising headline, but after reading the 1,200 words I'm none the wiser on what they built or whether it meets their own specs. A bit of a disappointment.
You might as well say "we saved 100% of cloud costs by writing our own cloud".
I use managed databases, but is there really that much to do for maintaining a database? The host requires some level of maintenance - changing disks, updating the host operating system, failover during downtime for machine repair, etc. if you use a database built for failover I imagine much of this doesn’t actually affect the operations that much assuming you slightly over provision.
For a database alone I think the work needed to maintain is greatly exaggerated. That being said I still think it’s more than using a managed database, which is why my company still does so.
In this case though, an append log seems pretty simple imo. Better to self host.
Maybe Im cynical but interesting that "the business" didnt start to check it to cut costs. I know that customers love this feature. Cynically I can see it costing more, so some customers would drop in.
Also it looks they rewrote a log / timeseries "database" / key value store? As pthers mention sounds like reinventing the wheel to get a cool blog post and boost career solving "problems".
Reminds me how I implemented mssql active-active log replication over dropbox shares back in 2010 to synchronise two databases in the Us and in the UK. Worked perfectly fine except of that one hurricane that took them out for longer than 14 days. This was more than the preconfigured log retention period.
the ebs slas look reasonable to a non expert like me and you can take snapshots. it sound like you need to be careful when snapshotting to avoid inconsistencies if stuff is only partially flushed to disk. so you'd need to pause io while it snapshots if those inconsistencies matter. that sounds bad and would encourage you to take less frequent snapshots...? you also pay for the snapshot storage but i guess you wouldn't need to keep many. i like that aws defines "SnapshotAPIUnits" to describe how you get charged for the api calls.
with aurora, it looks like you can synchronously replicate to a secondary (or multiple secondaries) across azs in a single region. it sounds nice to have a sync copy of stuff that people are using. op says the'yre ok with a few seconds of data loss so i'm wondering how painful losing a volume right before taking a snapshot would be.
i wonder if anything off the shelf does something similar. it sounds like people are suggesting clickhouse. i saw buffer table in their docs and it sounds similar https://clickhouse.com/docs/en/engines/table-engines/special.... it looks like it has stuff to use s3 as cold storage too. i even see geo types and functions in the docs. i've never used clickhouse so i don't know if i'm understanding what i read, but it sounds like you could do something similar to whats described in the post with clickhouse if the existing geo types + functions work and you are too lazy to roll something yourself.
And if you write your own db as they did here, it can 100% take advantage of your setup.
- This is core to their platform, makes sense to fit it closely to their use cases
- They didn't need most of what a full database offers - they're "just" logging
- They know the tradeoffs and designed appropriately to accept those to keep costs down
I'm a big believer in building on top of the solved problems in the world, but it's also completely okay to build shit. That used to be what this industry did, and now it seems to have shifted in the direction of like 5-10% of large players invent shit and open source it, and the other 90-95% are just stitching together things they didn't build in infrastructure that they don't own or operate, to produce the latest CRUD app. And hell, that's not bad either, it's pretty much my job. But it's also occasionally nice to see someone build to their spec and save a few dollars. It's a good reminder that costs matter, particularly when money isn't free and incinerating endless piles of it chasing a (successful) public exit is no longer the norm.
I get the arguments that developer time isn't free, but neither is running AWS managed services, despite the name. And they didn't really build a general purpose database, they built a much simpler logger for their use case to replace a database. I'd be surprised if they hired someone additional to build this, and if they did, I'd guess (knowing absolutely nothing) that the added dev spends 80% of their time doing other things. It's not like they launched a datacenter. They just built the software and run it on cheaper AWS services versus paying AWS extra for the more complex product.
