undefined | Better HN

0 pointsjsmthrowaway11y ago0 comments

And the point being made (and with which I agree) is there's probably fewer than a few hundred of such deployments in the world, and their problems are very likely not your problems.

I have good context on this from seeing the analytics warehouse for a very popular social network. Even with a multi-terabyte hot database, the analytics dataset fit on a single 2U server full of drives. I have not seen a deployment of Hadoop yet that makes sense. Usually I see things like a 2 or 3 TB log spread across 60 nodes and a job to count entries matching a regular expression (because Hadoop becomes the hammer hunting for nails), which can easily be done in a VM on a Mac in minutes.

That was the thrust of the original post, and I agree wholeheartedly.

0 comments

2 comments · 2 top-level

rakoo11y ago

If the data already lives on 60 nodes, then transferring all this data to a vm + performing work may be longer (and clog the pipes) than just doing the work where the data already is and aggregating results.

shmerl11y ago

> because Hadoop becomes the hammer hunting for nails

Yes, I agree that it's not an uncommon situation. But regardless whether legitimately big deployments are common or not, I never really appreciated some negligence with performance in using Java for such things. Just because it scales, it doesn't mean that overhead should be ignored, because the cost of the overhead scales too.

So going back to my original point, having such kind of systems made with Rust would be really much better.

j / k navigate · click thread line to collapse