undefined | Better HN

0 pointsMrPowers2y ago0 comments

Modern query engines are designed to read data in parallel because it's so much faster. The data could be stored in 50 different one-gig files that were read in parallel.

0 comments

jandrewrogers2y ago

None of this requires multiple files, either in theory or in practice. It may be a limitation of a particular implementation but that is a design choice. There are functional advantages to not splitting up files unnecessarily when processing queries.

theLiminator2y ago

Spec compliant CSV parsing cannot be done in parallel iirc. Most CSV parsers are not spec compliant, but splitting up CSVs is one way to read them in parallel while ensuring you can still use a spec compliant parser.

j / k navigate · click thread line to collapse

0 pointsMrPowers2y ago0 comments

Modern query engines are designed to read data in parallel because it's so much faster. The data could be stored in 50 different one-gig files that were read in parallel.

0 comments

jandrewrogers2y ago

theLiminator2y ago

j / k navigate · click thread line to collapse