Ho! And just to tease you a little bit. We highly improved the indexing speed of Meilisearch v1.6 (january 24). Here is a teasing tweet [2].
[1]: https://blog.meilisearch.com/best-practices-for-faster-index... [2]: https://x.com/kerollmops/status/1734576622303404317
What do you think about the blog posts content?
I want to have something running local and bundled into an app for friends, maybe more.
Interesting. Could you elaborate on the benefit of this?
I've (possibly prematurely) discarded the notion of using RoaringBitmaps like this because while they use less memory, traversal is and mutation is so much slower using a fixed buffer out of a pool and considering an upper bounded slice of putative results at a time.
Although this is for a search engine that typically deals with under-specified queries and is designed for best-effort retrieval given an upper computation time.
I don't know on what I can elaborate.
Storing integers that are near each other is much more optimal in a RoaringBitmap than in a flat array. The reason is that it will only store the integers by storing the high part once and the low part in an array or bitmap efficiently.
Also we already use RoaringBitmaps on the other end of Meilisearch and converting that to another data structure could take a lot of times.
Don't take this as me questioning your design process or choices, I'm sure they are well motivated, I just think it's a good practice to be curious as to how when someone reaches a different conclusion than yourself :-)
https://www.meilisearch.com/docs/learn/advanced/storage#meas...
so I never even tried Meilisearch as our base json is 5gb
