I've been a heavy Julia user for +4 years and adore this ecosystem. I use Julia for parallel computing, modeling and solving large-scale optimization problems, stochastic simulations, etc. During the last year or so, creating plots and dashboards has become much easier too.
Julia makes it surprisingly easy to go from "idea" to "large-scale simulation". I've used it in production and just for prototyping/research. I can engage with Julia as deeply as I would with C code or as "lightly" as I engage with Matlab/R.
I'm excited to see what comes next.
Extra excited that the project I happen to work on (the Parca open source project[2]) influenced this change [3][4]. Shout out to Valentin Churavy for driving this on the Julia front!
[1] https://github.com/JuliaLang/julia/commit/06d4cf072db24ca6df...
I provide the option of Julia in my tutorials. Students are lazy, and don’t want to explore something new. Most of them stick with matlab.
What prevents matlab users from switching? The syntax is similar.
Choosing a programming language based on just comparing the language syntax only works for academic settings or toy projects for self-curiosity and learning. Once you consider adopting a language for complicated real-world industry usage, you have to look beyond the syntax and compare ecosystem to ecosystem.
E.g. Look over the following MATLAB "toolboxes" and add-ons developed over decades: https://www.mathworks.com/products.html
Julia doesn't have a strong equivalent ecosystem for many of those. In that MATLAB product list is Simulink. Tesla uses that tool to optimize their cars: https://www.mathworks.com/company/newsletters/articles/using...
You can take a look at some of the 1-minute overview videos to get a sense of MATLAB toolboxes that companies pay extra money for: https://www.youtube.com/results?search_query=discover+matlab...
It has add-ons such as medical imaging toolkit, wireless communications (antenna signal modeling), etc. And MATLAB continues releasing new enhancements that the Julia ecosystem doesn't keep up with.
If one doesn't need any of the productivity tools that MATLAB provides, Julia becomes a more realistic choice.
Or to put it another way, companies didn't really "choose the MATLAB programming language". What they really did was choose the MATLAB visual IDE and toolkits -- which incidentally had the MATLAB programming language.
In the past I used Calca [3] for these kinds of things, and there are many of these "fancy calculator apps" around, but it's just so much nicer to work in a real programming language.
[1] https://painterqubits.github.io/Unitful.jl/stable/
[2] https://docs.julialang.org/en/v1/manual/unicode-input/
[3] http://calca.io/
It's got a frustrating "not fun" on-boarding. ie. the number of minutes from downloading "Julia" to getting cool satisfying results
1. It not a calculator on steroids like Matlab. It doesn't have one main open source IDE like Octave/Rstudio that you can drop in and play around in (see plots docs repl workspace)
2. The default language is more like a "proper programming language". To even make a basic plot you need to import one of a dozen plotting libraries (which requires learning how libraries and importing works - boring ..) and how is someone just getting started to decide which one..? I don't need that analysis paralysis when I'm just getting started
3. Documentation .. Well it's very hard to compete with Matlab here - but the website is not as confidence inducing. The landing page is a wall of text: https://docs.julialang.org/en/v1/ Tbh, from the subsequent manual listing it's not even clear it's a math-focused programming language . It's talking about constructors, data types, ffi, stack traces, networking etc etc.
1. I run Julia on my smartphone and often use it as calculator.
2. You typically only need Plots.jl for most needs. See https://docs.juliaplots.org/stable/
3. See https://juliaacademy.com
Another alternative environment are Pluto notebooks. It's reactive like a spreadsheet, but easy to use in your browser.
I have several users without much coding experience using Pluto notebooks just to generate plots from CSV files. They are finding the combination of a web based interface, reactive UI, and fast execution easier to use than a MATLAB Live script.
It's looking like VS Code (julia-vscode.org) will be the equivalent, and it's gotten a good chunk of the way towards that - "plots docs repl" are all existing features and pretty easy to use. The docs [1] show a "Workspace" feature too. There's also some integration with tools like JET.jl [2] so that there's in-editor code analysis and diagnostics.
(And the extension works in VS Codium as well, so you can go completely FOSS if that's your wish.)
[1] https://www.julia-vscode.org/docs/stable/userguide/grid/ [2] https://aviatesk.github.io/JET.jl/stable/
It’s true that the majority of the Julia community comes from a scientific or engineering background, and this is reflected in the package ecosystem and some of the “embedded” language features for technical computing. However I feel like stressing Julia as a “technical language” any more than it currently is may scare away people that may want to use it for other purposes, such as a general purpose scripting language.
Now more than ever before, with the improved code caching in 1.9, Julia lends itself perfectly for interactive usage and day-to-day scripting.
Today is not a few years back.
No one genuinely wants a review of the product from a few years back if they are seriously considering the language today.
Julia is not Lisp or Scheme.
I’ll offer my perspective. I’m switching little-by-little. Why don’t I just rip off the bandaid and switch whole-hog immediately? The literally twenty years of tools I’ve developed in MATLAB.
At work, I’m paid to perform analyses, develop algorithms, and document that work to my colleagues and our sponsors. I’m not paid to learn Julia. If I’m working on something completely novel where none of the MATLAB building blocks I’ve wrote over the years are useful, or the porting time is a small (for some arbitrary and subjective definition) of the overall task time, then I’ll do it in Julia.
The toolboxes aren’t a huge sticking point for me: Mathworks has only somewhat recently developed toolboxes geared toward my particular domain, so none of my code relies on them. My ability to share with colleagues is a bit of a sticking point. We’re predominantly a MATLAB shop (and this seems true not just at my company, but in our particular niche in industry). There has been some movement toward Python. But if it’s anything like the transition from Fortran to MATLAB — which was still on-going when I started in the early 2000s — then a full switch to either Python or Julia is still a ways off.
I'm very comfortable in Matlab and often know immediately what's wrong when I hit those oddities. In Python it usually means I spend considerable time googling and tinkering before I even understand what I did wrong because I have far less experience... Same when I tinker with Julia.
And for some people it being a Real Programming Language may be a disadvantage actually... Typically means you need a better understanding than just type-and-run.
I have made a career in python data science since then overall it was a very good decision for me. I was ahead of the curve when data science popped up.
I have since joined a company with engineering dept that rely on matlab. I have no doubt that they wouldn’t get much benefit from switching to python or Julia apart from the license costs.
I also did this and only the students that were well ahead of the rest gave it a try. But I think this is natural, many students are already pushed to the limit to manage the current set of studies. Learning a whole new language on top of that introduces a lot of extra risk and has an opportunity cost that might not make it feasible (unless you are already doing very well).
I don't think it is so much that students are lazy but more that the current way that the study plan is made in universities doesn't allow for much risk taking by the students. So they will just go the "cookie cutter" way.
It's not possible to understand why someone would prefer Matlab through a software engineer's lense. Our jobs are different and a real programming language isn't anywhere on our list of needs or wants for matrix laboratory work. We actually prefer semantics like copy-on-write and features like dynamically adding fields to structs at runtime. It fits our workflow better, and that's ultimately what matters most.
I'm sure one day I'll add Julia to the list of real programming languages that I've used to write a library, but I'll still wrap it and call it from Matlab just like everything else.
Julia as a compiled language is faster and more distributable than either, but there is a chicken-egg problem about the ecosystem. MathWorks' provided libraries for Matlab are excellent, amazingly documented and massively supported. Python libraries are just hugely numerous in any domain you can imagine...
So some other method e.g. Galpaca distilled or Llama-based distil by step for Matlab->Julia translation model should be popularized.
Especially with the distill step by step you could get something that runs quite efficiently on a laptop.
Pretty much everything students do is new to to them.
It’s still a completely different language with it’s own paradigms, semantics, ecosystem, etc.
To be clear, Julia constantly updates and I'm sure many of its original dependencies are being recoded to help address this issue. I still think it's worth a check and audit if someone wants to put together a broader project as to not get burned by this issue later.
Julia code is mostly distributed as source code, and compiling into standalone executables and then distributing them to outsiders is not very common, so it doesn't often come up as an issue. But it's an important detail to keep in mind in case the use case does arise.
On technical merits Matlab still comes ahead in fast plotting of large datasets. And Julia still has a reputation for a "go fast and break things" community and the corresponding cavalier response to bugs. Those will slowly change, but as of now, Matlab holds an advantage there. My 2c.
To all Julia users: Go forth, and make use of PrecompileTools.jl in your packages! The latency only drops if you actually make use of precompilation, and it's pretty easy to use. I can't wait for more of the ecosystem to start making use of it.
----------------------------
JULIA 1.8.5
julia> @time using Plots
11.341913 seconds (14.83 M allocations: 948.442 MiB, 6.88% gc time, 12.73% compilation time: 62% of which was recompilation)
julia> @time plot(sin.(0:0.01:π))
3.342452 seconds (8.93 M allocations: 472.925 MiB, 4.44% gc time, 99.78% compilation time: 78% of which was recompilation)
-----------------------------------
JULIA 1.9.0
julia> @time using Plots;
2.907620 seconds (3.43 M allocations: 195.045 MiB, 7.52% gc time, 5.61% compilation time: 93% of which was recompilation)
julia> @time plot(sin.(0:0.01:π))
0.395429 seconds (907.48 k allocations: 59.422 MiB, 98.54% compilation time: 74% of which was recompilation)
With the steady progress in improving precompilation, I'm optimistic to use it more often in the future, though.
When latency is much better and/or it can compile static binaries, the use case of Julia will hopefully broaden
Looks like this release reduces that by a lot, see the first section in the OP on caching native code, modulo adoption of good precompilation habits by the various packages.
However, because of this, when somebody asks me if I would recommend Julia to them, instead of answering “no” I just say “it depends”
Therefore, I‘m really excited for the improvements in code caching! Thanks to Tim Holy, Jameson Nash, Valentin Churavy, and others for your work
> 35 minutes to compile
What kind of CPU are we talking about here!?
I have similar compilation times with Rust on an old Asus 1215B, where 8GB and SSD hardly help the compile the world from scratch cargo model, when starting a new project.
It sounds like the issue is probably unrelated to using Pluto, and likely more to do with the streaming libraries used and memory management - but that's just a guess based on the minimal info here. When you say it couldn't handle streaming data, what issues did you have? By "streaming data with Pluto notebooks on the web" do you mean PlutoSliderServer or something else?
FWIW, Fons and co are very responsive to user issues (for eg. on the Zulip pluto channel [1]), so if you haven't tried that already, I'd recommend that. Similarly with Stipple, I believe they're trying to build a company out of it, so they'll probably be very receptive to business use cases and making them work.
[1] https://julialang.zulipchat.com/#narrow/stream/243342-pluto....
On the Stipple front it's a pacing issue, we think. The developers have been WONDERFULLY helpful and improved our code tremendously. GENIE is a great solution and I am sure that they will be successful. I believe they WILL produce a MWE of this task and we'll certainly look to see if we can make it work. Right now we can't as some of our lab equipment and financial systems generate sub 0.5 sec data stream.
I'm asking because I work on Shiny, a reactive web framework for Python (and R) that aims to solve this problem well, and I'm having trouble figuring out how Python people have been doing this sort of thing. It's straightforward with a lower-level web framework like Flask/FastAPI but then you lose the nice reactive properties of something like Stipple.jl (and Shiny).
We use zeroMQ to move data around so to get it into a python script is very easy for us. We "were" going to create a dash/plotly app to consume the data stream and create a trading augmentation tool as we find dash a pleasure to work with. We also find that it's very well supported using videos so not even we ( non web developers) can mess it up. We adhere to the Tufte approach to graphical representation of complex data sets and python allows us to take a minimalist approach to doing that.
We are also discussing the implementation of async components in a python specific hosting company that seems to address the same market as GENIE. That would give us the GUI approach should we chose to use it. We have experienced some problems with streaming data into Stipple but I would expect that they will provide a MWE that we can modify. I have high hopes for the GENIE team, they seem like excellent people.
All that said we are progressing out Carbon discussions with various people. In Chicago we are very lucky that we have a deep pool of people skilled at kernel latency avoidance, this helps us consider what we can expect from Carbon and if there are any advantages to what we want to achieve.
I hope this answers your question.
That's big! Now I can add packages to my startup.jl without having to worry that every single REPL startup will be slowed down by them. This also eases the pain of things being moved away from the standard library, since we can just add them back to the base environment and load them at startup, making it basically the same thing.
* `Alt-e` now opens the current input in an editor. The content (if modified) will be executed upon exiting the editor
* A "numbered prompt" mode which prints numbers for each input and output and stores evaluated results in Out can be activated with REPL.numbered_prompt!() (basically `In[3]` `Out[3]` markers like in Mathematica/Jupyter).
> set the env var `JULIA_PKG_PRESERVE_TIERED_INSTALLED` to true.
How is this different from setting `Pkg.offline(true)` and then doing the `add`? I don't know the intricacies of how it works, but that's what I've been doing when I just need to try something out in a temp environment.
I'd assumed that global fastmath was a bad idea in general, and assumed that was the reason for making this a no-op. Is there a reason it's particularly bad in Julia, some assumptions the standard library makes or something?
It makes sense to optimize for the non-fast math case because that's the recommended setting, and I guess having two implementations of all the (very important, easy to mess up, core) special functions + all the testing infra to check that they work correctly on all platforms was probably deemed too much work for marginal benefits.
I was just wondering why the post said "impossible to use correctly in Julia" rather than just "impossible to use correctly", but writing it out now, I realize that "impossible" would be hyperbole for the latter, it would be more like "highly likely to go wrong".
There are still some other difficulties of course, since fastmath in the C ABI is quite wild (or I guess, it's really the GCC implementation up to GCC 13 (https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55522#c45)). Simon wrote a nice piece about the difficulties in general: https://simonbyrne.github.io/notes/fastmath/. In a general sense there is still the potential vulnerability that effects the Python ecosystem which is that if any package has binaries built with fastmath it could cause other calculations to be fastmath as well in a non-local way (https://moyix.blogspot.com/2022/09/someones-been-messing-wit...):
> It turns out (somewhat insanely) that when -ffast-math is enabled, the compiler will link in a constructor that sets the FTZ/DAZ flags whenever the library is loaded — even on shared libraries, which means that any application that loads that library will have its floating point behavior changed for the whole process. And -Ofast, which sounds appealingly like a "make my program go fast" flag, automatically enables -ffast-math, so some projects may unwittingly turn it on without realizing the implications.
With Julia, there is the advantage here that (a) most libraries don't have binary artifacts being built in another language, (b) the Julia core math library is written in Julia and is thus not a shared library effected by this, and (c) those that do have their binaries built and hosted in https://github.com/JuliaPackaging/Yggdrasil. So in the binary building and delivery system you can see there are some patches that forcibly remove fastmath from the binaries being built to avoid this problem (https://github.com/search?q=repo%3AJuliaPackaging%2FYggdrasi...). The part (b) of course is the part that is then made globally safe by the removal of the flag in Julia itself, so generally Julia should be well-guarded against this kind of issue with these sets of safeguards in place.
does it mean I still have to invoke special workflows and commands to get compilation benefits or does it work out of the box for normal julia invocations?
The tradeoffs are somewhat larger load times (TTL), increased precompilation time (because some of the compilation moves to precompile time), and increased disk usage by the package.
The post says "TTL has also been reduced, albeit not as dramatically as TTFX." And the graph seems to indicate the same. Is that not true, or are you comparing it to pre-1.7 TTLs (which are not shown in the post), or is it just context(/project)-dependent?
Also, I wish Julia was as popular in Europe as it is overseas.
I am curious what are the fields where it is less well developed?
Data engineering and cloud integration is a big one. It has very few tools in that domain, and I say this as a heavy Julia user (hobby).
What is a "Julia situation" and how is a programming language a cult? It's used by companies, programmers, scientists, etc. to do stuff. This is a strange take.
People invest huge amount of effort fixing Python's shortcomings (pyspark, tf, jax, mojo) requiring a completely different way of thinking modulo the syntax. And nobody is talking about the "cult of the snake"
Sorry, pretty shallow complaint. Great work!
Things of course only show up in the HN front page when they reach a sexy conclusion, which also means that what shows up on HN is a very biased subset of the discussion which omits most subtlety and posts the biggest speedup numbers. Most of the day-to-day of course is things more 10% changes in some case, where only when compounded 100 times you finally have a story the general HN public cares to hear. This also generally means that the long discussions of caveats and edge cases is also filtered from what most of the public tends to generally read (it's just difficult to capture some things in a blog post in any concise way), so if you care for the nuance I highly recommend joining some of the Slack channels.
I know this doesn't inform you about dev work on Julia in general, but it goes into detail with the recent improvements to latency
Are both of those still true? I'm a zero-index guy, but having index offsets is fine as long as the standard library is high quality. As for LLVM, I'd prefer it not need a fork but that's less important.
The main problem is non-standard library packages that were written back in early julia days before OffsetArrays existed (e.g. a big offendeder IIRC was StatsBase.jl), and so wasn't written with any awareness of how to deal with generic indexing.
OffsetArrays.jl are a neat trick, and sometimes they really are useful e.g. when mimicing some code that was written in a 0-based language, or just when you're working with array offsets a lot, but I wouldn't really recommend using them everywhere. Other non-array indexable types like Tuple don't have 0-based counterparts (as far as I'm aware), so you'll be jumping back and forth from 0-based and 1-based still, and it's just an extra layer of mental load.
Honestly though, it's often not very necessary to talk about array indices at all. The preferred pattern is just to use `for i in eachindex(A)`, `A[begin]`, `A[end]` etc.
> and IIRC also the language build depends on a fork of LLVM (https://github.com/JuliaLang/llvm-project)
Yes, we use a fork of LLVM, but not because we're really changing it's functionality, just because we have patches for bugs. The bugs are typically reported upstream and our patches are contributed, but the feedback loop is slow enough that it's easiest to just maintain our own patched fork. We do keep it updated though (this release brings us up to v14) and there shouldn't be any divergences from upsteam other than the bugfixes as far as I'm aware
Can the same be done with Firefox's `about:memory`'s `Load...` button, or is it Chromium specific?
