DevOps in Academic Research (opens in new tab)

mhm... having skimmed through the article, the only reason (imho) is not wanting to wait for hpc/slurm to schedule your jobs... all the other stuff, could have been done with university resources. They usually offer some shared drive space and are often forthcoming in supporting special needs (at least from my experience).

Having used both AWS Spot Instances for compute jobs, as well as hpc resources (slurm), the amount of work to getting it run is comparable... Except that the aws stuff quickly gets expensive when needing thousands of cpu hours per "experiment", and money is usually tight at universities ;-). Some ansible scripts automate basically everything away for myself. But using the tool you know is probably not always a bad idea.

Thanks! AWS was costly compared to the free HPC cluster but it was sub $1000 per month and our bottlenecks were person-hours rather than money. We had a lot of work to do. As some have mentioned there wasn't a strong technical argument for switching to AWS, it was more about productivity with those tools. I'd built up a bit of credibility with the team at that point and was like "I can make this better" and my boss was said "ok cool". As I mentioned in the article the team was very open to try new things.

I can see an alternate timeline where I set up Buildkite in AWS to run Slurm jobs on the HPC cluster, rather than using EC2 spot instances. Using Buildkite + S3 were probably the more important infra changes.

A lot of companies are donating computing power for COVID-19 research, so it's possible they have a deal of some kind worked out with Amazon.

> That is my biggest hurdle working at a university right now.

We took the opposite approach and assembled our own equipment, which we self host and manage. The cost savings we're significant, and what we lose doing admin and maintenance is saved by having the flexibility and freedom to code and run what we want, when we want.

> wonder how he convinced the team to drop the "for free" academic HPC/Slurm

Probably because they are working with publicly available data so they don't need to worry about HIPAA/restricted data compliance. One of the main reasons for using the HPC/Slurm cluster is that IT (or the data compliance office) guarantees compliance with restricted data.

I had a case where using the "for free" academic HPC cluster was so inconvenient that we used only in very few cases when we needed very large amounts of compute, and for any smaller tasks using the commercial cloud solutions saved enough time/salaries so that they were worthwhile even purely from cost perspective. Also, for academic purposes it was possible to get quite large amounts of cloud computing for free, so most users never bothered to even try to approach the academic HPC environment as it had no advantages whatsoever.