undefined | Better HN

0 pointsorthoxerox1y ago0 comments

I was comparing it with Parquet, which is much more complex, but has features that help you access the data in less than O(n), like row groups and pages.

0 comments

cm21871y ago

you mentioned NLJSON and CSV, which would require to read all columns from the disk.

orthoxeroxOP1y ago

Yes, but you would usually have to read at least two columns anyway. What are the datasets that are too large to be ingested completely, but too small for a proper columnar format?

If ZSV is meant to occupy the gap between CSV/NLJSON (smaller datasets) and Parquet/DuckDB (larger datasets), this niche is actually really small, if not nonexistent.

cm21871y ago

yes it's unclear to me what is the advantage over parquet with compression. And there are enough file formats flying around already.

j / k navigate · click thread line to collapse

0 pointsorthoxerox1y ago0 comments

I was comparing it with Parquet, which is much more complex, but has features that help you access the data in less than O(n), like row groups and pages.

0 comments

cm21871y ago

you mentioned NLJSON and CSV, which would require to read all columns from the disk.

orthoxeroxOP1y ago

Yes, but you would usually have to read at least two columns anyway. What are the datasets that are too large to be ingested completely, but too small for a proper columnar format?

If ZSV is meant to occupy the gap between CSV/NLJSON (smaller datasets) and Parquet/DuckDB (larger datasets), this niche is actually really small, if not nonexistent.

cm21871y ago

yes it's unclear to me what is the advantage over parquet with compression. And there are enough file formats flying around already.

j / k navigate · click thread line to collapse