A Walk with LuaJIT (opens in new tab)

(polarsignals.com)

152 pointsdamir1y ago26 comments

26 comments

I wrote the code and the blog, happy to answer any questions/comments. Very eager to have folks try it out and give feedback! Like is my meme game strong or very strong? J/K

There's some missing bits around FFI and callbacks (i.e. C calling function pointer that is a luajit generated stub back into the interpreter) and curious if anyone actually uses these things in OpenResty workloads. Deploy and enjoy!

aktau1y ago

I'd love to use this to profile Neovim, where (nowadays) user scripts are often written in Lua 5.1. Neovim by default embeds LuaJIT (close to HEAD). Being able to see Lua functions and C functions in the stack would be amazing.

Is there a single-shot mode that I can use from the command-line? Something like:

    $ parca-perf record nvim ...

Or:

    a$ nvim ...
    b$ parca-perf top -p $(pidof nvim)

brancz1y ago

There isn’t, because the indented way to use Parca is to profile production and always-on.

However, we wouldn’t be against adding a mode like this!

FWIW both the server and the agent are single statically linked binaries so while it’s a bit more set up it’s not terribly difficult either[1].

[1] https://www.parca.dev/docs/quickstart/

gnurizen1y ago

It should work to profile neovim, does it use the luajit shared library or is it statically linked? You could run parca-agent and set up a filter to discard all non-neovim samples and leave a parca instance running locally and just restart it periodically.

1 more reply

brancz1y ago

Thanks for submitting! We know HN has a sweet spot for LuaJIT, so we figured it would eventually end up here.

Quick summary: this post dives into the gory details of how we implemented an eBPF based profiler for LuaJIT.

Let us know if you have any questions on this, we’ll keep an eye out on comments!

neomantra1y ago

Very deep dive, thank you for sharing it all. So cool it traverses callbacks too.

brancz1y ago

Glad you liked it! Yeah, we worked with a customer who really needs this badly and has done some unspeakable things to get by until now.

alberth1y ago

I’m tremendous excited about LuaJIT 3.0 development.

https://github.com/LuaJIT/LuaJIT/issues/1092

Q: does anyone know timeline on the release?

rurban1y ago

Looks like 10 years to me

slekker1y ago

Why do you think that?

1 more reply

benwilber01y ago

LuaJIT.org stopped publishing release tarballs [1] which caused leafo's GH actions builds [2] to suddenly stop working. The workaround was to start testing against OpenResty's distribution of LuaJIT [3] which is incompatible with LuaJIT.org's version.

There is no faster way to make a fork the de facto standard version than to break everyone's CI builds.

[1] https://luajit.org/download.html

[2] https://github.com/leafo/gh-actions-lua/issues/49

[3] https://github.com/openresty/luajit2

krapp1y ago

The workaround for LuaJIT moving to Github was to... clone a fork of it?

If they could do that, why can't they just pull from the LuaJIT repo?

tecleandor1y ago

LuaJIT didn't move to GitHub, they just have a mirror there.

The thing is they stopped numbering and publishing releases, it's all a rolling release without any name or number, so you cannot snapshot I'm certain version.

But OpenResty fork does create tag versions with date, so they can build or test against certain concrete snapshot frozen in time.

1 more reply

mraleph1y ago

At some point in my life (when I briefly worked on LuaJIT for DeepMind) I have written a stack walker which can stitch together native and Lua frames: for each native stack frame it checks if that is actually an interpreter frame or a trace frame - if that's the case it finds corresponding `lua_State` and unwinds corresponding Lua stack, then continues with native stack again.

This way you get a stack trace which contains all Lua and native frames. You can use it when profiling and you can use it to print hybrid stack traces when your binary crashes.

I was considering open-sourcing it, but it requires a bunch of patches in LJ internals so I gave up on that idea.

(There is also some amount of over-engineering involved, e.g. to compute unwinding information for interpreter code I run an abstract interpretation on its implementation and annotate interpreter code range with information on whether it is safe or unsafe to try unwinding at a specific pc inside the interpreter. I could have just done this by hand - but did not want to maintain it between LJ versions)

dreampeppers991y ago

Lua and Nginx are fantastic. Did you know it's possible to add behavior/code lua for openrest dynamically? https://github.com/leandromoreira/lua-resty-dynacode?tab=rea...

j / k navigate · click thread line to collapse

26 comments

gnurizen1y ago

I wrote the code and the blog, happy to answer any questions/comments. Very eager to have folks try it out and give feedback! Like is my meme game strong or very strong? J/K

aktau1y ago

Is there a single-shot mode that I can use from the command-line? Something like:

    $ parca-perf record nvim ...

Or:

    a$ nvim ...
    b$ parca-perf top -p $(pidof nvim)

brancz1y ago

There isn’t, because the indented way to use Parca is to profile production and always-on.

However, we wouldn’t be against adding a mode like this!

FWIW both the server and the agent are single statically linked binaries so while it’s a bit more set up it’s not terribly difficult either[1].

[1] https://www.parca.dev/docs/quickstart/

gnurizen1y ago

1 more reply

brancz1y ago

Thanks for submitting! We know HN has a sweet spot for LuaJIT, so we figured it would eventually end up here.

Quick summary: this post dives into the gory details of how we implemented an eBPF based profiler for LuaJIT.

Let us know if you have any questions on this, we’ll keep an eye out on comments!

neomantra1y ago

Very deep dive, thank you for sharing it all. So cool it traverses callbacks too.

brancz1y ago

Glad you liked it! Yeah, we worked with a customer who really needs this badly and has done some unspeakable things to get by until now.

alberth1y ago

I’m tremendous excited about LuaJIT 3.0 development.

https://github.com/LuaJIT/LuaJIT/issues/1092

Q: does anyone know timeline on the release?

rurban1y ago

Looks like 10 years to me

slekker1y ago

Why do you think that?

1 more reply

benwilber01y ago

There is no faster way to make a fork the de facto standard version than to break everyone's CI builds.

[1] https://luajit.org/download.html

[2] https://github.com/leafo/gh-actions-lua/issues/49

[3] https://github.com/openresty/luajit2

krapp1y ago

The workaround for LuaJIT moving to Github was to... clone a fork of it?

If they could do that, why can't they just pull from the LuaJIT repo?

tecleandor1y ago

LuaJIT didn't move to GitHub, they just have a mirror there.

The thing is they stopped numbering and publishing releases, it's all a rolling release without any name or number, so you cannot snapshot I'm certain version.

But OpenResty fork does create tag versions with date, so they can build or test against certain concrete snapshot frozen in time.

1 more reply

mraleph1y ago

This way you get a stack trace which contains all Lua and native frames. You can use it when profiling and you can use it to print hybrid stack traces when your binary crashes.

I was considering open-sourcing it, but it requires a bunch of patches in LJ internals so I gave up on that idea.

dreampeppers991y ago

Lua and Nginx are fantastic. Did you know it's possible to add behavior/code lua for openrest dynamically? https://github.com/leandromoreira/lua-resty-dynacode?tab=rea...

j / k navigate · click thread line to collapse