[1] https://github.com/toshi-search/Toshi
"Native" or not has nothing to do with the garbage collector. Go is a GCed language too, but I really don't think it's as performant or tuneable as Java's.
https://itnext.io/go-does-not-need-a-java-style-gc-ac99b8d26...
Once you put aside all the words and see how things stack up in the real world, Go and Java are pretty comparable, usually with the JVM having a slight edge after the JIT is nice and warm. They go about it in different ways, but there's nothing wrong with that.
But basically, go can get away with a less advanced GC due to having and relying on value types quite often. But Java and certain workloads do require the state-of-the-art GCs that the JVM provides. And a search engine likely constitutes such a one.
Elasticsearch restarting due to out of memory errors was a common occurrence at my last gig where we used it heavily.
The difference between Java GC and Go GC is stark. In several years, I've never had to touch a GC knob on Go, but in Java it is expected, and not usually worth the effort.
The default G1 GC is perfectly fine for the vast majority of workloads as is. In the rare case it is needed, the target pause time setting can be used to swift our preference of latency vs throughput.
In that regard, a Java runtime compiled in AOT form is hardly any different from a Go runtime.
As for the lack of value types, the solution is on the way, and apparently many keep forgetting that writing a JNI based library is much better than throw everything way and start from scratch in random language X.
RAM isn't a bottleneck on most servers these days. Even the cheapest EC2 tends to have 2GB which will run nearly any Java app fine.
Sure, Rust is better. But it will take years to replicate what's already out there in Java.
I'll take a database written in Java over C any day given equivalent choices.
The Rust based search engine Tantivy is handily faster than Lucene and supports pretty much the same features.
It isn't hard to scale, but it is the most expensive thing to scale.
I know it's not equivalent, but Redis is written in C and it's amazing what you can do with it. It is also incredibly efficient and what it does.
I don't get the hate towards C. Is it the security issues? (These exist in Java too you know).
Java is as safe as Rust. It's just 1/2 the speed and uses 2-3X the ram. It was the best C alternative for DB until a couple years ago.
I still think Java has a lot of value to bring or to learn and use for any programmer.
The ecosystem of libraries, software, tooling, cross platform nature and with the advent of GraalVM - Java is still very alive and capable !
As for JVM guys - Kotlin seems be an easy migration.
BTW - what do you think about dart ?
You do not need Rust necessarily. Mixed feelings yet. I am not big fan. Golang is good too
Disclaimer: I'm a cofounder.
In terms of performance you see a lot of comments that are stating X is faster than Y. I'd take such comments with a large grain of salt. Unless those lightweight alternatives actually do the same things you can't really compare the performance. It's not that hard to make indexing fast in Elasticsearch if you disable all the features that make it slower. Of course if you don't actually have those features it's going to be fast by default because it simply isn't doing anywhere close to the same things.
Elasticsearch is actually pretty damn fast even when you do use its many features. The reason is that it relies on in memory caches, thread pools, etc. and that a lot of very smart people have been implementing and optimizing very efficient algorithms in Lucene for the last 25 years. Elasticsearch actually can run with as little as 256MB but of course you are not going to be able to cache a lot of data with that and performance will suffer accordingly. Mostly large heap sizes with Elasticsearch are all about using larger caches. It also relies on memory mapped files and OS file caches for that. That's what allows it to work with extremely large data sets and still provide query responses in milliseconds.
There's no reason you wouldn't be able to do the same with a native implementation of course. But it would be naive to assume you'd end up using a lot less memory or CPU. Using that would be pretty core to matching performance and features. Not using that would make it pretty hard to get even close.
My experience with ES has been somewhat mixed, exactly because them trying to do everything at the same time and a lot of the specific things we needed being under documented and involving much more try and error than should be necessary.
I for one welcome more focused products, when the need again arises for something in this space.
Unused classes can be unloaded from RAM and it's like they don't exist. The executable is still huge but lower memory and CPU cache footprint.
ES is truly focusing on metrics and things and does have some features to make those use cases easier that Solr would probably need a lot of configuration/customization for.
So, Solr is about search. ES is about a specific set of use cases that rely very heavily on search.
And Lucidworks Fusion (commercial alternative to ES) is about big data and ML and full multi-tool pipeline on top of Solr.
Lovely, but the missing features is basically what makes Elasticsearch great!
Missing features:
Clustering and High Availability
The distributed systems and related constraints and guarantees is what takes the resources !
( I would assume he would use a system based on raft to offer the above, or using something like Infinispan, helix or hazelcast kinda )
1. I was hoping for a drop-in replacement for Elasticsearch. A new/different API means Zinc can't leverage existing tools that use Elasticsearch.
2. I don't like that you're bundling zinc with a UI; that disadvantages anyone else trying to build a better UI and often (usually?) leads to tying the db too closely to the UI (or vice versa)
If it were written in Node.js or PHP, on the other hand, you'd know to stay ten miles away.
People here seem to assume Elasticsearch requires lots of memory and CPU. It actually scales down very nicely in addition to its famous ability to scale up very well. Any decent developer laptop should have no issues running this. Running VS Code is more of a burden on my laptop. And intellij is even worse.
/s