Kubernetes chart which runs with StatefulSet so compute time would not be lost when node is going down and/or pods are being rescheduled:

https://github.com/pcktdmp/charts

This design pattern is extremely useful when running for instance on AWS Spot Instances.