Show HN: Fast `git status` (git core.fsmonitor in Rust) (opens in new tab)

(github.com)

52 pointsjgavris8y ago18 comments

18 comments

18 comments · 7 top-level

bluejekyll8y ago· 5 in thread

How does this differ from the standard fsmonitor?

This appears to mainly be a lightweight wrapper of watchman.

In my benchmarks it's about twice as fast as the reference Perl script monitor. Also just a quick exploration for me into Rust.

https://github.com/git/git/blob/v2.16.0/templates/hooks--fsm...

glandium8y ago

You should make it clearer that the benchmark is perl fsmonitor vs. rust fsmonitor. Without reading the comments here, I would have thought it was without fsmonitor vs. with.

jgavrisOP8y ago

Good suggestion! Without the fsmonitor enabled at all that looks more like 400ms on my iMac Pro.

bluejekyll8y ago

Thanks for the link. Yes, that makes sense. It’s also less code. :)

Edit: well, maybe just fewer comments... ha

jgavrisOP8y ago

Yes, the Perl is very short, and has more comments. I missed the Perl era so it was a bit of work for me to figure out how it worked enough to implement it in Rust.

sigi458y ago· 2 in thread

At first i thought i don't understand what your little rust code is doing, than i got it.

Its a watchman wrapper written in rust.

But in that article you linked, there is mentioned that git ships with a sample watchman hook already?!

jgavrisOP8y ago

It is not enabled by default yet, since it depends on an external tool to provide repo file update events. Git has one bundled that uses Facebook's watchman (with homebrew if you've installed it like so), but you have to configure each repo (or globally) to enable it. I was just curious how much faster a Rust implementation could be over a Perl script, and wanted to learn a bit of Rust.

Phrodo_008y ago

I wonder how much faster it would be to use the OS' inotify equivalent (if Linux, inotify itself) directly from rust rather than using an external tool.

pitaj8y ago· 2 in thread

I wonder what kind of gains could be seen by using a fully rust-native bundle with something like notify [1]

[1]: https://crates.io/crates/notify

avar8y ago

This is not at all comparable to what the OP submitted and would be at least a many thousands of lines worth of effort.

What watchman is doing is doing inotify-as-a-service for you, in order for any program to use the Linux inotify API it needs to be running constantly so it can keep consuming events from the kernel, and in the case of the use case watchman fulfills, maintain an internal database of how the current state differs from various times in the past.

There's lots of very nasty edge cases you need to deal with. E.g. if you don't consume the kernel events fast enough (or the buffer is too small) new events will be dropped.

You'll get told about this by the kernel, but now the only way to re-build your in-memory representation to match reality is to recursively walk the FS with readdir(), stat() etc, but at the same time consume new inotify events and update your in-memory structures while doing both.

Needless to say this is a lot more complex than just using watchman off the shelf.

It's also pointless to optimize this area, watchman usually returns in 2-3 milliseconds even on huge trees, but because of bugs / misfeatures in the current git-side implementation it can take hundreds of milliseconds until we make sense of all of that. This is because git doesn't really "know" about watchman/inotify, it just uses it as a replacement for walking the FS, but on big repositories other stuff can still be expensive.

To OP: I highly encourage you to submit this to the Git mailing list for comment. I've been involved in this area (helped write the Perl hook) and there's people who'd be interested in having this be e.g. a part of git's contrib/ directory so it can be compiled with git.

jgavrisOP8y ago

Thanks for the explanation, indeed watchman does all the heavy lifting! I will submit it to the mailing list, I'd love to see it in git core.

avar8y ago· 1 in thread

If you're optimizing for speed use BSER instead of JSON to talk to watchman: https://facebook.github.io/watchman/docs/bser.html

jgavrisOP8y ago

Fantastic suggestion! I will give this a go soon.

https://github.com/facebook/watchman/tree/master/rust/serde_...

mwean8y ago· 1 in thread

What about using entr[1] instead? I've had great success using it for rebuilding ctags on file changes.

[1]: https://github.com/clibs/entr

jgavrisOP8y ago

I have a 'git aware' bash prompt, so my primary use case is making sure my prompt is ready with the git status really fast.

benpeart8y ago

Very cool! I wrote the original (with help) in Perl as it has the advantage of being installed with git but I love the speed advantage of your Rust implementation. Also helps shows the advantage of making core.fsmonitor a pluggable interface.

ufmace8y ago

Cool idea, but I tried it on one of my team's bigger repos, and it's actually slower than stock.

j / k navigate · click thread line to collapse

18 comments

18 comments · 7 top-level

bluejekyll8y ago· 5 in thread

How does this differ from the standard fsmonitor?

This appears to mainly be a lightweight wrapper of watchman.

jgavrisOP8y ago

In my benchmarks it's about twice as fast as the reference Perl script monitor. Also just a quick exploration for me into Rust.

https://github.com/git/git/blob/v2.16.0/templates/hooks--fsm...

glandium8y ago

You should make it clearer that the benchmark is perl fsmonitor vs. rust fsmonitor. Without reading the comments here, I would have thought it was without fsmonitor vs. with.

jgavrisOP8y ago

Good suggestion! Without the fsmonitor enabled at all that looks more like 400ms on my iMac Pro.

bluejekyll8y ago

Thanks for the link. Yes, that makes sense. It’s also less code. :)

Edit: well, maybe just fewer comments... ha

jgavrisOP8y ago

Yes, the Perl is very short, and has more comments. I missed the Perl era so it was a bit of work for me to figure out how it worked enough to implement it in Rust.

sigi458y ago· 2 in thread

At first i thought i don't understand what your little rust code is doing, than i got it.

Its a watchman wrapper written in rust.

But in that article you linked, there is mentioned that git ships with a sample watchman hook already?!

jgavrisOP8y ago

Phrodo_008y ago

I wonder how much faster it would be to use the OS' inotify equivalent (if Linux, inotify itself) directly from rust rather than using an external tool.

pitaj8y ago· 2 in thread

I wonder what kind of gains could be seen by using a fully rust-native bundle with something like notify [1]

[1]: https://crates.io/crates/notify

avar8y ago

This is not at all comparable to what the OP submitted and would be at least a many thousands of lines worth of effort.

There's lots of very nasty edge cases you need to deal with. E.g. if you don't consume the kernel events fast enough (or the buffer is too small) new events will be dropped.

Needless to say this is a lot more complex than just using watchman off the shelf.

jgavrisOP8y ago

Thanks for the explanation, indeed watchman does all the heavy lifting! I will submit it to the mailing list, I'd love to see it in git core.

avar8y ago· 1 in thread

If you're optimizing for speed use BSER instead of JSON to talk to watchman: https://facebook.github.io/watchman/docs/bser.html

jgavrisOP8y ago

Fantastic suggestion! I will give this a go soon.

https://github.com/facebook/watchman/tree/master/rust/serde_...

mwean8y ago· 1 in thread

What about using entr[1] instead? I've had great success using it for rebuilding ctags on file changes.

[1]: https://github.com/clibs/entr

jgavrisOP8y ago

I have a 'git aware' bash prompt, so my primary use case is making sure my prompt is ready with the git status really fast.

benpeart8y ago

ufmace8y ago

Cool idea, but I tried it on one of my team's bigger repos, and it's actually slower than stock.

j / k navigate · click thread line to collapse