undefined | Better HN

0 pointstheamk5y ago0 comments

Do use CSV (and other similar formats) for read-only data which fits entirely in the memory.

It is great for data safety -- chown/chmod the file, and you can be sure your scripts won't touch this. And if you are accessing live instance, you can be pretty sure that you won't accidentally break it by obtaining a database lock.

Now "csv" in particular is kinda bad because it never got standardized, so if you you have complex data (punctuation, newlines, etc..), it you might not be able to get the same data back using a different software.

So consider some other storage formats -- there are tons. Like TSV (tab-separated-values) if you want it simple; json if you want great tooling support; jsonlines if you want to use json tools with old-school Unix tools as well; protobufs if you like schemas and speed; numpy's npy if you have millions of fixed-width records; and so on...

There is no need to bother with SQL if the app will immediately load every row into memory and work with native objects.

0 comments

4 comments · 2 top-level

daniellarusso5y ago· 2 in thread

I agree with the JSON suggestion, but what advantage is there to TSV versus CSV?

I have experienced pain with both characters (tab and comma), particularly when I am not the one creating the output file.

junon5y ago

Tabs do not appear in common literature. They're easier to justify inputs not have them in order to avoid having a quotation or escaping mess.

Commas are _way_ too common.

CSV is an awful format anyway.

theamkOP5y ago

If you can make sure the data has no newlines or tabs, the TSV needs no quoting. It is just "split" function, which is present in every language and very easy to use. When I use it, I usually add a check to writer that there is no newlines or tabs in data, and assert if this is not the case.

You use this tsv with Unix tools like "cut", "paste", and ad-hoc scripts.

There is also "tsv" as defined by excel, which has quoting and stuff. It is basically a dialect of CSV (Python even uses the same module to read it), and has all the disadvantages of CSV. Avoid it.

oblio5y ago

> complex data (punctuation, newlines, etc..)

Oh, the irony! Text with punctuation and newlines is complex data.

CSV is doomed but the world runs on it and pays its cost with engineer tears.

j / k navigate · click thread line to collapse