undefined | Better HN

0 pointspixelmonkey11y ago0 comments

You said: "they should have defined their schema upfront".

We tried. It was not only less disk efficient, but also slower for queries. Wasn't the result we expected, but alas.

0 comments

3 comments · 1 top-level

nemothekid11y ago· 2 in thread

If thats the case, then thats something you should have highlighted that as well. Now that I think about it we decided to store blobs under one column using msgpack for a similar reason (although mine was "we had mongo eat up all our disk space because of field names, so now our code is littered with single character field names"). I've thought about fixing it, but if its the case I guess thats related to CASSANDRA-4175. The map solution makes it come across you weren't quite sure what you were doing.

However, I'm even more surprised at the "slower for queries" part. Maybe I'll do some tests with COMPACT STORAGE.

pixelmonkeyOP11y ago

Might be specific to the Python driver, but it was slower for queries because most of the cputime was being spent decoding column structure/metadata and on type conversions. We also tested msgpack blobs, iirc, but the xsv format was the winner for query perf and for compactness on disk.

Would have gladly gone into more detail, but at 4,000+ words for the blog post already... :)

nemothekid11y ago

Hmm, so I tried converting a table we had from a standard table to one with COMPACT STORAGE. The space saving wasn't all that great (100GB -> 80GB), not near 30x.

I stand by my point that the mistake was using CQL maps when you should have just used a defined schema.

j / k navigate · click thread line to collapse