undefined | Better HN

0 pointsxiaodai7y ago0 comments

Given Julia has macros, so it will definitely catch up to R and data.table in terms of syntax (if it's not already there). I am more thinking about performance, e.g see https://h2oai.github.io/db-benchmark/. It shows that Julia is lagging behind on group-by (and from my experience many other operations) when compared to R's data.table.

Although I have done some work to make thing fast see: https://github.com/xiaodaigh/FastGroupBy.jl. I have yet to update it to Julia v1. Hopefully, I will get to that soon. However, the improvement I have made only works for grouping up to 2 group-by variables and I need to learn more about generated functions to make the code more generic. So from my (someone who's actually spent time trying to optimise these data operations) perspective, Julia will take a while to catch up. Hats off to the data.table crew!

0 comments

3 comments · 1 top-level

byt1437y ago· 2 in thread

Ah yes, that is true. However Julia is tackling a harder problem in that the speed lag is presumably due to optimizing for custom element and table types.

xiaodaiOP7y ago

"for custom element and table types" that is highly likely to be true, but unless an equally fast Julia program exists I remained scientifically skeptical. But my prior believe is that Julia can be as fast.

ChrisRackauckas7y ago

DataFrames.jl doesn't parameterize on its types and tries to rely on function barriers to be fast enough. That's what gives it the speed issue. IMO this isn't the best idea. It's not difficult to write a data table that's easier to optimize than that, but you would have to give up some of the flexibility.

j / k navigate · click thread line to collapse