undefined | Better HN

0 pointsFridgeSeal7y ago0 comments

Write speed is fine, it’s more the fact that the dataset is reasonably large, and to run an instance with enough capacity and nodes (even with spill to disk), is silly expensive.

0 comments

mverwijs7y ago

Being in a simular situation what do you consider "reasonably large"?

FridgeSealOP7y ago

250-300GB.

Not large by absolute standards sure, but large enough to cause issues.

I’m sure there’s some kind of solution that involves re-architecting the ES cluster and indices and re-architecting the data flows and stuff. But if our options are go through all that, or seriously slim down our architecture and costs by just running Sonic + our data warehouse, I’m definitely going to give it a go. After all, worst comes to worst we can go down the re-architecting ES route if Sonic doesn’t work out.

¯\_(ツ)_/¯

Xylakant7y ago

I’d be curious what your expectations and constraints are, but from my experience of running clusters in the double digit TB-Size my ballpark figure for that amount of data would be 2 medium size data nodes and a small tiebreaker. Alternatively, if you can live with the reduced resilience and availability, even a single node might just do. Depends on the expectated churn though, ES really does not like document updates.

3 more replies

j / k navigate · click thread line to collapse