I'm talking more about sqlite itself than any given product around it at this point, but I still think it's an interesting thought experiment in this context.
This write latency might be fine, although more than one backend app I know renewed the expiry time of a user session on every hit and would thus do at least one DB write per HTTP call. I don't think this is optimal, but it does happen and simply going "well don't do write traffic then" does not always line up with how apps are actually built. Replicated sqlite over litestream is very cool, but definitely you need to build your app around and also definitely something that costs you one of your innovation tokens.
The point of the post is just that if you can get rid of most of the big operational problems with using server-side SQLite in a distributed application --- most notably, failing over and snapshotting --- then SQLite can occupy a much more interesting role in your stack than it's conventionally been assigned. SQLite has some very attractive properties that have been largely ignored because people assume they won't be able to scale it out and manage it. Well, you can scale it out and manage it. Now you've got an extremely simple database layer that's easy to reason about, doesn't require you to run a database server (or even a cache server) next to all your app instances, and happens to be extraordinarily fast.
Maybe it doesn't make sense for your app? There are probably lots of apps that really want Postgres and not SQLite. But the architecture we're proposing is one people historically haven't even considered. Now, they should.
I'm not sure "litestream replicate <file>" really costs a whole innovation token. It's just SQLite. You should get an innovation rebate for using it. :)
I think this offering, and this idea, are absolutely fantastic, and if not the future, at least a big part of it, for the reason outlined in the post: namely, that for a lot of apps and use cases, sqlite is more than enough.
But I also suspect this is probably already the case, and we don't know about it because people don't talk about it.
Amusingly, I was recently scolded here on HN for suggesting to use sqlite, by someone who said HN was a place for "professionals":
https://news.ycombinator.com/item?id=29908921
Once sqlite is synced to a virtually indestructible storage solution (or several!} then that removes a lot of objections from "professionals".
Could not be more excited by this. Congrats.
I’m coming at this from the perspective of a traditional django app that has .using(“write”) when wanting to write data. Otherwise you’re replaying requests at a completely different app server.
Am I understanding this correctly?
That being said, you don't get an innovation rebate for using a new tool, even if, as here, it's a parsimony-enabler. It's still a new tool.
A description from TFA reads "The most important thing you should understand about Litestream is that it's just SQLite." (This reminds me an awful lot of the tagline for CoffeeScript: "It's just JavaScript" -- where did that leave us?) But that info box is just under a description of how the new tool is implemented in a way that makes it sound (to someone who's never looked at the SQLite codebase) like it's breaking some assumptions that SQLite is making. That's the sound of an innovation token being spent.
Well, no, because every insert will still be fast (until there's too many). It does not block until it's written to e.g. s3.
So there's a window, let's say 1 second, of potential data loss.
I assume syncing the wal to s3 is much faster than inserting to sqlite, so it will never fall behind, but I have not tested.
> Also, every company I ever worked at had a policy to run at least two instances of a service in case of hardware failure.
Yeah, but the goal is not to have X instances, the goal is to limit downtime. In my experience the complicated setups have downtime as well, often related to how complicated they are.
In my mind a setup like this would only be used where some downtime is OK. But that's quite common.
