Interesting. Do you know which version of Splunk you were using? Our latest version has vastly improved our query performance over large datasets.

Re. an open source log indexer: Agree, this is a space that will eventually become dominated by open source tools, used particularly by startups and small businesses. I think most people ignore this use of a MapReduce-like framework because they conceptually understand how it could be used, but 99% of all work is in the implementation, not the idea. And as of yet, I don't believe there has been a specific implementation beyond what companies like Shopify are doing where they add nice GUI tools on top of awk and grep (which admittedly is probably good enough for most people / business on this forum).

For scalability use SenSage (http://www.sensage.com/customers). Their some of their customers gather 200GB-500GB of log data daily and at least one customer has a petabyte of historical log data under management. Query performance over long time ranges is important ... when you discover a new exploit in use, how can you tell how long it's been employed unless you can look at history?

Yes, Everest is cool stuff.

If you have huge datasets, we'd love to hear from you. If even to chat for a few minutes about what your data looks like. Ping me at info@drawntoscale.com -- maybe we can help!