undefined | Better HN

0 pointssitkack1y ago0 comments

I switched to JSONL over a decade ago and I would recommend everyone else to also have switched then.

This whole thread is an uninformed rehash of bad ideas.

0 comments

I think that might make sense ingest side, but that's very expensive to deal with if you're doing anything remotely large.

I think sinking into something like delta-lake or iceberg probably makes sense at scale.

But yeah, I definitely agree that CSV is not great.

sitkackOP1y ago

JSONL as a replacement for CSV, you shouldn't be using CSV as format for long term storage or querying, it has so many downsides and nearly zero upsides.

JSONL when compressed with zstd, most of "expensive if large" disappears as well.

Generating and consuming JSONL can easily be in the GB/s range.

theLiminator1y ago

I mean on the querying side. Parquet's ability to skip rowgroups and even pages, and paired with iceberg or delta can make the difference between being able to run your queries at all versus needing to scale up dramatically.

1 more reply

j / k navigate · click thread line to collapse

0 comments

theLiminator1y ago

I think that might make sense ingest side, but that's very expensive to deal with if you're doing anything remotely large.

I think sinking into something like delta-lake or iceberg probably makes sense at scale.

But yeah, I definitely agree that CSV is not great.

sitkackOP1y ago

JSONL as a replacement for CSV, you shouldn't be using CSV as format for long term storage or querying, it has so many downsides and nearly zero upsides.

JSONL when compressed with zstd, most of "expensive if large" disappears as well.

Generating and consuming JSONL can easily be in the GB/s range.

theLiminator1y ago

1 more reply

j / k navigate · click thread line to collapse