On the public DDN (data.splitgraph.com:5432), we enforce a (currently arbitrary) 10k row limit on responses. You can construct multiple queries using LIMIT and OFFSET, or you can run a local Splitgraph engine without a limit. We also have a private beta program if you want a managed or self-hosted cloud deployment with the full catalog and DDN features. And we are planning to ship some "export to..." type workflows for exporting to CSV and potentially other formats.

For live/external data, we proxy the query to the data source, so there is no theoretical data size limit except for any defined by the upstream.

For snapshotted data, we store the data as fragments in object storage. Any size limit depends on the machine where Splitgraph's Postgres engine is running, and how you choose to materialize the data when downloading it from object storage. You can "check out" an entire image to materialize it locally, at which point it will be like any other Postgres schema. Or you can use "layered querying" which will return a result set while only materializing the fragments necessary to answer the query.

Regarding ClickHouse, you could watch this presentation [0] my co-founder Artjoms gave at a recent ClickHouse meet-up on the topic of your question. We also have specific documentation for using the ClickHouse ODBC client with the DDN [1], as well as an example reference implementation. [2]

[0] https://www.youtube.com/watch?v=44CDs7hJTho

[1] https://www.splitgraph.com/connect

[2] https://github.com/splitgraph/splitgraph/tree/master/example...

We support both! A Splitgraph repository can "mount" an external data source, and we'll proxy queries to it using a system based on Postgres Foreign Data Wrappers (FDW). But a repository can also contain any number of "data images," which are versioned snapshots of data roughly inspired by Docker images. You can define them with a declarative, Dockerfile-like syntax called a Splitfile, and you can rebuild them against upstream sources with caching semantics similar to "docker build."

Our core philosophy has always been that it makes sense to start with data federation (live data), and then selectively warehouse/ingest only what you need (versioned data). We're shipping some upcoming features to support this workflow. You start by providing us (or your private deployment) a set of read-only credentials to any supported data source, which we then "mount" as a repository, making it discoverable in the catalog, and instantly queryable with all the other data on Splitgraph. If or when you decide that you want to warehouse this data, we'll make it easy for you to schedule a loading job to ingest it as a Splitgraph image. This way, you can query the live or versioned data in any repository, by simply changing the tag you use to address it.

You can do all this stuff locally, btw – a decentralized workflow is fully supported, and you can push data between peers. The public Splitgraph.com happens to be a "super peer" with a data catalog, scalability features, etc. But if you just want to experiment on your own, you can try it in five minutes!

I don't want to spam this thread too much, so I'll limit it to one link – maybe take a look at the Splitfile docs: https://www.splitgraph.com/docs/working-with-data/using-spli...