undefined | Better HN

0 pointsmalux858y ago0 comments

short answer: not really. longer answer: depending on how old the systems are, they're likely not useful for learning modern distributed computing.

For example, a friend of mine had four 4 core, 8GB machines and tried to configure a hortonworks cluster, but the system spent so much time just running the cluster state, that you could only do trivial computing on top of the cluster before the nodes started timing out.

He then spent ages disabling unnecessary services, re configuring components to be more memory efficient - and spent so long doing all of this maintenance he never actually learnt how to compute on top of the cluster, which was his original goal.

So it's best just to pay a few hundred a month to amazon and run your own small cluster that you can spin up and down as you need

0 pointsmalux858y ago0 comments

short answer: not really. longer answer: depending on how old the systems are, they're likely not useful for learning modern distributed computing.

So it's best just to pay a few hundred a month to amazon and run your own small cluster that you can spin up and down as you need

0 comments

3 comments · 1 top-level

stephenr8y ago· 2 in thread

FYI Hadoop is not a synonym for “cluster”.

A clustered system could be a load balancer and a few servers running percona cluster, or a couple of ldap masters with read-only relicas and proxy-writes.

Doing it on bare metal also gives exposure to the parts amazon specifically don’t

malux85OP8y ago

Yes I didn't mean to imply that, only that it can be challenging dealing with a resource constrained environment.

For example our deep learning clusters do not use hadoop, but use multi-process python daemons (written by me) and communicate with protocol buffers. Writing these high performance daemons myself has taught me a lot about cluster computing and some of the tradeoffs that have to be made in order to keep the GPU pipelines full.

For anyone else reading: Pyro is quite fun if you're in a limited environment and want to learn https://pythonhosted.org/Pyro4/ Also, setting up a spark https://spark.apache.org/ environment can be done on smaller machines too

stephenr8y ago

Right, but I think you still missed my point.

The vast majority of software developers will interact with, if it’s running clustered it’s because the project needs higher availability for that service (typically the database) and potentially higher performance in production.

1 more reply

j / k navigate · click thread line to collapse