undefined | Better HN

0 pointsim_down_w_otp10y ago0 comments

You could do a lot of fast number crunching if you wanted to. It just gets tricky to implement without screwing with the primary scheduler pool.

I've written many NIFs for Erlang that call out to high-performance C routines when "number crunching" is most important. I've also done it to side-step memory allocator churn for implementing binary protocols that support zero-copy semantics like Capnp by using the NIF as a kind of escape hatch where I can fiddle bits in a buffer directly.

In fact, somewhat lazily since it's low priority, I'm trying to create a decent Rust wrapper for the NIF interface so that I can attempt to get more compile time safety out of the dangerous stuff I do outside BEAM's safety net.

Other than being dangerous, since a segfault will bring down the whole VM, the other big issue is these calls block the scheduler (basically breaking the VM's preemptive abilities and creating the same limitations Akka on the JVM has when you block all the threads in an ActorSystem). However, now that R18 has dirty-scheduler support it's much easier to avoid this problem by quarantining that stuff on threads that don't muck with executing normal bytecode.

0 comments

lostcolony10y ago

You're not using Erlang/Elixir for number crunching; you're using it for orchestration of your C code.

That's my point; in a niche language, you only try to use it to solve the problems it claims or you have reason to believe it to be good at, thus, its limitations (both declared and any you run into outside of the claims it makes) are dismissed with "you're using the wrong tool", a fact the developer usually realizes themselves, and does not treat as a deficiency of the language.

If you had reason to believe Erlang was good at number crunching, and wrote code to crunch numbers in it, and then found it wasn't, you'd complain about how slow it is. Because you never had that expectation, because Erlang flat out says it's not good at that, you knew to instead write NIFs. The thought "this language should be faster" never entered your mind, because that kind of performance is not the goal of the language. Instead you used it where its reliability and scalability come into play, things it is good at, and offloaded the number crunching to a language better suited for it.

im_down_w_otpOP10y ago

That's a fair point. Though the thought, "this language should be faster" has crossed my mind many times during the pain of all that other mucking about I had to do :-)

I'm timidly hopeful that projects like ErLLVM and BEAMJIT will eventually produce enough improvement to BEAM performance for computational workloads that using FFI escape hatches only becomes necessary in the most extreme fringe of circumstances.

The fact that the NIF interface is so damned straight forward to work with compared to FFI implementations in other systems does ease the pain a bit of having to step out to consume it more often than I might its analogue in other languages.

j / k navigate · click thread line to collapse