Instead we have a bajillion layers of CRUD all in slightly different protocols just to do the same read or write to the database.
This breaks down quickly once you have data that could become private or mutate rather than append.
The "one db per user" model for private data made using other features like views etc more difficult when you have to upgrade,edit,remove them.
Mutability wasn't really a problem, either present the conflicts to user and pick one or write code to merge if possible.
Take a look at GraphQL, its central promise is to let client choose what's the optimal data it needs (that often denormalized through nested GraphQL queries), and send it in one batch.
It is not to say there shouldn't be a simple replica. It is just if we want it to be a simple replica, we should have a server-side mirrored some-what-denormalized representation rather than just the raw server-data models.
If immutable fact/datom streams with idealized cache infrastructure becomes a thing (and architecturally i hope it does) it's going to need DRM to be accepted by both users and businesses.
As soon as your server state is larger than whatever your client can handle, the whole metaphor breaks down.
In 2015, my business implemented a Meteor-based real-time vehicle tracking app utilising Blaze, Iron Router, DDP, Pub/Sub
Our Meteor app runs 24hrs/day and handles hundreds of drivers tracking in every few seconds whilst publishing real-time updates and reports to many connected clients. Yes, this means Pub/Sub and DDP.
This is easily being handled by a single Node.js process on a commodity Linux server consuming a fraction of a single core’s available CPU power during peak periods, using only several hundred megabytes of RAM.
How was this achieved?
We chose to use Meteor with MySQL instead of MongoDB. When using the Meteor MySQL package, reactivity is triggered by the MySQL binary log instead of the MongoDB oplog. The MySQL package provides finer-grained control over reactivity by allowing you to provide your own custom trigger functions.
Accordingly, we put a lot of thought into our MySQL schema design and coded our custom trigger functions to be selective as possible to prevent SQL queries from being needlessly executed and wasting CPU, IO and network bandwidth by publishing redundant updates to the client.
In terms of scalability in general, are we limited to a single Node.js process? Absolutely not - we use Nginx to terminate the connection from the client and spread the load across multiple Node.js processes. Similarly, MySQL master-slave replication allows us to spread the load across a cluster of servers.
For those using MongoDB, a Meteor package named RedisOplog provides improved scalability with the assistance of Redis's pub/sub functionality.
PouchDB is mentioned a couple of times, including one “PouchDB compatible” mention. Wondering what unique use cases RxDB supports?
I mean, that's why it didn't catch on right :-) It's hard :-)
The "shard" could just be that users own feed in this case. Then you get offline for free where user adds a tweet and it appears immediately, replicating back to server when he goes back online. The server replica side will need to be a lot more complicated to deal with broadcasting but I don't see why it won't work.
If I had to make a Twitter clone with CouchDB, I would probably have one timeline document per user, and maybe one per day to limit the syncing bandwidth.
It's still in very early stages (although I am using it in production for my company)
It's very similar to Debezium (mentioned in another comment), but it's built with Phoenix (elixir), so great for listening via websockets.
Basically the Phoenix server listens to PostgreSQL's replication functionality and converts the byte stream into JSON which it then broadcasts over websockets. This is great since you can scale the Phoenix servers without any additional load on your DB. Also it doesn't require wal2json (a Postgres extension). The beauty of listening to the replication functionality is that you can make changes to your database from anywhere - your api, directly in the DB, via a console etc - and you will still receive the changes via Phoenix.
I still have to document a lot of how it works and how to use it, but if anyone is interested then I will make it a priority over the weekend
Is there any mileage in doing this with triggers? I have a _very_ legacy system which needs caching adding. Rather than dig through the code to invalidate the cache every time a record is updated/deleted in 20+ tables, I am thinking that being able to listen to the SQL executed and invalidate the cache based on the tables involved would be a clean approach.
But not found any way to make that possible - yet.
1. Triggers have an 8000 byte limit. I ran against these limits pretty quickly
2. You need to attach the trigger each time you create a new table. With this you can set and forget
I'm using gRPC in other languages/frameworks and I'd really like to migrate one of those to Elixir in the near future, but I'm curious to know if anyone else has given it a try.
There's a prominent warning on the repo:
This project looks really awesome, and very much has the shape we wished Horizon could have turned into
Is there still maintenance?
https://github.com/rethinkdb/rethinkdb/issues/6747#issuecomm...
TLDR: They are hoping to announce a revival in the near future.
I have certainly created infinite loops while using React's componentDidUpdate, maybe it's just important to define triggers on single attributes rather than entire database rows.
https://docs.hasura.io/1.0/graphql/manual/subscriptions/inde...
For anyone who wants something similar that's not GraphQL, then I'm in the early stages of developing a Phoenix (Elixir) implementation which broadcasts changes over websockets: https://github.com/supabase/realtime
See comments here: https://news.ycombinator.com/item?id=21354039
Demo vid https://v.usetapes.com/85knuAPruB Longer description in github readme https://github.com/jtmarmon/hackerthreads
Your example is similar to RethinkDB and others. A websocket streaming json.
[1]: https://github.com/tonsky/datascript
See also Google Wave, which had "real time" as its main novelty, but ultimately failed.
It is run by Internet Archive, HackerNoon, DTube, Notabug, etc.
Handles about 8M monthly active users!
+ end-to-end encryption, graphQL & graph data, upcoming Svelte support, decentralized, etc.
MIT/Zlib/Apache2 licensed.
or, even, "what is gun replacing?"
I mean, it looks interesting and I would like to play around with it, but I don't understand enough to know what kind of stuff I can build with it.
I also tried to understand the codebase which is impossible with obfuscated code like this one: https://github.com/amark/gun/blob/master/lib/store.js#L16
I've used it in some projects and found it useful. Thanks.
I noticed an instant downvote. It's not the first time i see this behavior on rxdb comments.
RxDB does exactly one thing which is being a client side database. You are not tied to a specific ecosystem or backend database.
Anyway, database is hard, opensource is great. We love them both, RxDB or Meteor.
This is based on meteors oplog-observe-driver https://github.com/meteor/docs/blob/version-NEXT/long-form/o...
So basically when a change-event comes, RxDB does not run the query against the database again, but instead uses the old results together with the event to calculate the new results.
It basically registers itself as a fake replica server so that it can get updates from the master (like binlog in case of MySQL) and then it forwards those updates to Kafka. The possibilities are endless what you can do with those updates as Kafka consumers.
I suppose this could be the killer feature to warrant using a new database -- because otherwise, from a user point of view, the result is the same.
EDIT: never mind, I should have read the article.
Another issue is Authorization and Authentication, I could not find a good solution for me for CouchDB. Couchbase seems to have better solutions for this but the premium plan seems really expensive and as far as I understood you need a "server" and a "sync server" which don't have low system requirements, at least for me.
Authentication is much easier when you use the GraphQL replication. There you are much more flexible on which data you return depending on which user is asking for it.
We used this for our mobile apps and the experience was pretty awesome. The ability to live observe both individual objects in the dB and results of queries makes building reactive UI’s a very pleasant experience.
I'm not doing field projects anymore but knowing this exists would have helped me tremendously in the last few years.
Thanks for sharing!
This is a nice feature for some, but not all situations..
Are there triggers that run on every update on the dataset?
