The one drawback ES had in the bad old days was that backup and restore was a nightmare... ESPECIALLY on AWS. The new system they introduced was so simple I was concerned about updating to it because I was SURE something would go south.
But it all just worked.
I still have the Couch to ES replication running because I'm anal like that... but really... yeah... you can do without Couchbase, Mongo et al... ES will probably do everything you need PLUS everything you can't do in the others.
also, i have the exact opposite nitpick. people want to use it to do everything, mail indexers, file system indexers. what's the matter with web developer folks? why is it that when the next database comes around they want to use it for everything?
I help run a large ES cluster (with canonical data in MySQL), and I consider this cautious attitude by the ES developers to be a good thing.
Elasticsearch is brilliant as a NoSQL - and if you were already using elasticsearch as a search system, you dont need to introduce yet another component into your stack.
Other than that (which is just performance tuning, really), ES matches mongodb feature for feature, and obviously has a lot of extra power from its search heritage such as facets and percolate.
So I can't actually think of any limitations, and it's why I said ES makes a better MongoDB than MongoDB.
You need to have a good understanding of how tokenizers and analyzers work to be able to create good results for your data. I have difficulties matching documents with the exact title being searched for. On MongoDB that just works, on ElasticSearch you need to configure it.
ElasticSearch has some advantages and MongoDB others. I think they are great together. One for storage and the other for searching.
http://docs.mongodb.org/manual/applications/geospatial-index...
You create a number of shards for each index(database) that you can't later expand.
It's search capabilities and scalability and fantastic - were throwing GB of data into it weekly and it just soaks it up.
That said, it's definitely worth looking into both, depending on what your needs are.
(IMHO) Unfortunately for most of the people, old habits to be made. Indeed a nice project and great release.
A few days before launch, things were not looking good. As admins manipulated articles in preparation for the launch, the servers kept crashing.
In a time-constrained major launch like this, a lot of nasty little hacks build up in the codebase. Our search system for admins was a complete mess. It was a custom solution that worked fine when admins managed a handful of database records, but now that they were managing thousands of articles, it was not scaling at all.
At the 11th hour, we dropped elasticsearch into our infrastructure. It worked like a charm. The servers stopped crapping out, and we launched on time.
Elasticsearch mostly "just works", and we didn't have to worry about complex schema definitions, working with giant complex XML files (hello Solr), or build anything on top to interface between the index and the queries themselves (Lucene). Thanks elasticsearch, you saved us!
If you were using Solr there are a few operational modes to run in. Config file based or SolrCloud[0]. The latter is more akin the ES in terms of cluster management.
I agree though from an simplicity of deployment perspective at scale ES is has a much lighter learning curve.
[0] https://cwiki.apache.org/confluence/display/solr/SolrCloud
`java -jar elasticsearch.jar` does a better job and that's basically all it takes. I'm planning to switch as soon as https://github.com/elasticsearch/elasticsearch/issues/256 lands.
Otherwise, we love ES. The other comment about it being a better Mongo than Mongo rings true. With the backup/restore API and the some of the circuit breakers, I'm hopeful that my fears will be abated.
Another example are disk-based doc values [2], which are essentially pre-computed field data structures that are stored on disk. This moves Field Data off heap and allows the OS to manage memory evictions, to help minimize GCs and OOM blowouts.
[1] http://www.elasticsearch.org/guide/en/elasticsearch/referenc...
[2] http://www.elasticsearch.org/blog/disk-based-field-data-a-k-...
Having supported Solr/ES/Lucene in production for 4+ years now (websolr.com / bonsai.io) I would be pretty hesitant to trust Lucene in general as a primary data store. Beautiful for secondary indexing, but otherwise, Why Not Postgres?™ ;)
EDIT: I don't think MongoDB is there yet either. There are definite benefits and drawbacks between Postgres and ES, tipping heavily towards Postgres for structured heavy write data. But for ES and MongoDB? I think MongoDB falls a bit short there.
What I'm doing is slamming the full text output of OCRed PDFs into a MyISAM table, the entire document in a text field.
What I'm afraid I'm not doing right is creating the web interface to search elasticsearch. What I'm using filters with the query string syntax[1] in the search box, pointing directly at that fulltext column. I'm also using the highlight functionality so that I can specify how many highlight blurbs to return with the result. The query string syntax works great with the OCR'd text, because most of it is near-garbage (as most ocr is) so you can search for something like "net sales"~50 to find those two terms within 50 words of each other. I think the results were something like: net sales 15,000 results "net sales" 120 results "net sales"~50 550 results
Can anyone point me at a good web based search implementation using elasticsearch that explains how they're doing it?
What I have works pretty good, I just want to... check my work, I guess.
[1]: http://www.elasticsearch.org/guide/en/elasticsearch/referenc...
The main thing for good stability and performance is to be very good at batching your updates. You don't want to sling a ton of highly-parallel single-document updates at Lucene, lest you thrash the JVM and start garbage collecting like crazy.
From there, on the query side, you'll want to get a good working knowledge of the different tokenization and analysis options. There are a lot of subtle and interesting combinations to be had in there that influence performance and relevance of your search results.
Do you mean that you use ES to do indexing on the backend of your documents and make it available on the web? Or do you mean that you use ES to index documents available on the web and let people to search for them?
I fetch a steady stream of FOIA documents, close to the maximum possible each week, and PDF/OCR them. I expose a web interface to the analysts I work with, to help them gather up documents for further analysis.
The second guess would probably be more interesting to most people.
Lucene is one of those projects which hardly has any real competition. That's surprising given how many real world software projects have a search requirement. While Lucene is excellent, it's not without flaws and competition is always great.
Solr, ElasticSearch, etc. are mostly concerned about the index/search features, and they do quite a good job there. But this still leaves a huge amount of space for commercial offerings, as core search is only a part of the problem. I'm thinking about connectivity with complex enterprise systems, support for the specific security models of those systems, integration in other systems, etc. Believe me, those problems are not easy to solve.
So, even if we have an index that can most probably match Lucene's feature for feature and quite a lot of things beside, we typically won't go after deals where simple search is the only requirement. Instead we focus on larger deals with more complex requirements. And we're doing quite well, thank you :)
What makes it so special to have hundreds of votes and tweets all around within 2 hours?
I don't understand. A DB engine engineer.
1. It handles human written language. Any language. The same technology that let's it handle strings written in human language provides a lot of flexibility in handling string in other applications. Particular when handling logs.
2. Non-string data it also handles very fast and cleanly (numbers, dates, geo).
3. Lucene has an inverted index that has been optimized over many years. ES scales that pretty seamlessly across many servers. All decisions in the project seem to be made around whether a feature can scale to 100s of nodes.
The devs have also been really smart to focus on the "out of box experience". Very well thought out defaults.
More on our experience with ES at scale: http://gibrown.wordpress.com/2014/01/09/scaling-elasticsearc...
https://lucene.apache.org/core/
"index size roughly 20-30% the size of text indexed"
That seems excessive for an index.
and many other stuff
Pick your favourite users group here: http://elasticsearch.meetup.com/
Full disclosure: I started and run the Berlin UG. We set ourselves apart by always providing a small introduction into ES for those that are completely new and would have a hard time following the main talk.
I don't see many tutorials covering usage of ES here: http://www.elasticsearch.org/tutorials/
Could you maybe provide a link to yours?
Yep, tutorials is a huge problem, but there are people working on that.
Congrats to the ElasticSearch team, and all the supporters around it. Once I get back into more of a coding role, I'll definitely be contributing back to the ES project.
There are other reasons, but that is like 90% of GC issues. To solve it, you need to make sure your faceted fields are configured well (usually not_analyzed) and assess how much memory is available. You may be able to index and even full-text search ten billion docs on a single machine, but faceting it may just be too much to ask for a single node.
Omiting norms, disabling bloom filters on old indices and enabling doc values are other ways to help alleviate field-data pressure.
Other GC culprits can be: too large bulk requests, unbounded threadpool queues, or something like parent/child/scripts/filter cache keys eating all your memory. Also don't go above 30gb heaps, the JVM becomes unhappy :)
1) Dremel clones [2] like Impala & Presto (for near real-time, ad hoc analytic queries over large datasets)
2) Lambda Architecture [3] systems (where queries are known up- front, but need to run against a large dataset)
Does anyone here have experience ES in such usecases, beyond the free text searching one ES is well-known for?
[1]: https://groups.google.com/forum/#!topic/elasticsearch/iTy9IY...
[2]: http://static.googleusercontent.com/media/research.google.co...
[3]: http://jameskinley.tumblr.com/post/37398560534/the-lambda-ar...
“Geo queries used to use miles as the default unit. And we
all know what happened at NASA because of that decision. The
new default unit is meters.”
ES adds distribution (multimaster-replicated cluster of nodes connected via a gossip protocol), sharding, defines a document model and schema (the mapping of arbitrary JSON documents to index structures), faceting, aggregation (ie., roll-up-type calculations), various types of scoring (eg., geographic distance), ETL ("rivers"), backup/restore, performance metrics, a plugin system (eg., for indexing different file formats) and a bunch of other things -- and of course a REST-based API on top of the whole thing.
At my last place of work, ES was beautiful and required little work to get a very fast, workable search in place.
For our current project we went with ElasticSearch and we're quite happy. One of the contributing factors was that one of our most experienced guys was unable to get the damn thing installed, even with the help of one Endeca consultant.
It's these little details I love, when a project actually cares about operations and not just "well here's the API"
I've been using ElasticSearch only for Logstash, but i've been blown away so far as how easy it is to deal with.
The github lays it out well.
"Sales data search: Writing a query parser / AST using pyparsing + elasticsearch"
Part 1: http://blog.close.io/sales-data-search-writing-a-query-parse...
Part 2: http://blog.close.io/sales-data-search-writing-a-query-parse...
https://groups.google.com/d/msg/elasticsearch/Rb7Lei4gaaE/7I...
Disclaimer: I’m the founder of a hosted Search As A Service and we use ES in a few critical parts of our infrastructure.
The rails support for it is amazing too. The guy creating the rails integration lib is really talented and active.
I can't recommend qbox.io enough! Point-and-click scaling of managed elasticsearch clusters + Kibana == bliss.