undefined | Better HN

0 pointsmej109y ago0 comments

This would be great, but I don't think it works on AWS -- I don't think they support broadcast.

0 comments

7 comments · 2 top-level

kchoudhu9y ago· 5 in thread

My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query.

Lots of ways to skin this cat...

vegabook9y ago

indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is completely IP address agnostic. Indeed you could run my version even with DHCP for all nodes.

kchoudhu9y ago

For sure. In my case, the bootstrap IP is distributed during initial software configuration. Not elegant as your solution, but it works.

As for the SPOF: my logic was that if the DB is dead, there's no point having the ZeroMQ components come up anyhow. The jobs will keep on trying to come up, but until Ops brings the database back up, they can't do any damage.

1 more reply

oldmanjay9y ago

What if they crash without registering an exit?

kchoudhu9y ago

Python context managers and heartbeats :)

vegabook9y ago

They broadcast every 2 seconds. No heartbeat = dead.

1 more reply

vegabook9y ago

That's very interesting and I did not know that. I happen to run my own modest (12-node) cluster, and had planned to move to cloud if I grew beyond 20 nodes. The fact that I would not own the transport layer in my case is quite severely problematic. Going to have to investigate if this is the case at Azure and the rest too. Can't I partition my instances into a VPN and UDP broadcast within that? If anybody has any insight, very happy to hear about it.

j / k navigate · click thread line to collapse

0 comments

7 comments · 2 top-level

kchoudhu9y ago· 5 in thread

My ZeroMQ components all register themselves in a database when coming and going. This makes it trivially easy to find where stuff is just by running a bog simple database query.

Lots of ways to skin this cat...

vegabook9y ago

indeed, with the small caveat that you now need to know the database's IP address, thereby introducing a failure point / master IP point. My solution is completely IP address agnostic. Indeed you could run my version even with DHCP for all nodes.

kchoudhu9y ago

For sure. In my case, the bootstrap IP is distributed during initial software configuration. Not elegant as your solution, but it works.

As for the SPOF: my logic was that if the DB is dead, there's no point having the ZeroMQ components come up anyhow. The jobs will keep on trying to come up, but until Ops brings the database back up, they can't do any damage.

1 more reply

oldmanjay9y ago

What if they crash without registering an exit?

kchoudhu9y ago

Python context managers and heartbeats :)

vegabook9y ago

They broadcast every 2 seconds. No heartbeat = dead.

1 more reply

vegabook9y ago

That's very interesting and I did not know that. I happen to run my own modest (12-node) cluster, and had planned to move to cloud if I grew beyond 20 nodes. The fact that I would not own the transport layer in my case is quite severely problematic. Going to have to investigate if this is the case at Azure and the rest too. Can't I partition my instances into a VPN and UDP broadcast within that? If anybody has any insight, very happy to hear about it.

j / k navigate · click thread line to collapse