undefined | Better HN

0 pointshpcjoe2y ago0 comments

My packages at work are about 1.5k lines. Precompilation is about 10-15 seconds. After precomp, about 1-2 seconds to load with "import" or "using". The longest I've seen on precomp have been DifferentialEquations.jl. I've not used it in a while, though I have plans to for personal (non-work related) projects.

My packages have 5-10 dependencies in them, I tend to keep my packages/tooling streamlined, and I performance optimize them (everything from loading time, through data structure/algorithm implementation) quite thoroughly.

Other users at my firm are adopting Julia as well. It isn't displacing python, though that is a possible future. Its similar enough that many grasp it right away. Its fast enough that its a viable alternative to Python + C++.

This said, Julia is not a silver bullet[1], though it is an excellent programming language. It has been able to do all of the same tasks as my python code. Faster (often multiple orders of magnitude) working with 10s to 100s of GB of data, in parallel.

Its a better solution for my workflow, and an increasing number of others. If its not to your liking, that's fine. No need to try to push it down with, from my vantage point, what seems like specious claims (very long precompilation times close to an hour, or so). If you have such actual examples, please post them. I'd love to see them. Chances are we could optimize this fairly easily.

[1] https://www.merriam-webster.com/dictionary/silver%20bullet

0 comments

cookieperson2y ago

Diffeqs precompilation is particularly heinous if you aren't on a serious machine. but I think I misspoke. I thought we were discussing package compilation. Unfortunately though, even if we are discussing precompilation theres a ton of cost there even if you go beyond packages. Time to first anything is often brutal. See the discussions on time to first gradient... This is often up to the developer to work out, but it's a challenge that's pretty unique to Julia itself.

When I see people describing viable alternatives to python and or C I personally look at C++ and Rust. Julia's GC is good for most academic embarrassingly parallel number crunching things, but is rough for large scale applications. I've only been able to use Julia in a vacuum for research. For product development, every effort I've seen has eroded insanely fast due to things other languages control much more easily. Those languages can often also do the math fast enough too, especially when the cost of failed experiments is accounted for. All those wait three hours to find out your first gradient descent iteration had a type error that propagated to 1000 compute nodes($) moments are gone. It just can't happen in other paradigms, and in some paradigms it's far less likely to happen and when it does the cost is minor because the cost of compilation was already amortized.

j / k navigate · click thread line to collapse

0 pointshpcjoe2y ago0 comments

[1] https://www.merriam-webster.com/dictionary/silver%20bullet

0 comments

cookieperson2y ago

j / k navigate · click thread line to collapse