> static IPs
FWIW, I personally love Virtual IPs (VIPs) for this (basically, an existing network interface advertises serving more than one IP, and can change that IP dynamically between servers with an arp call). The downside is that there are a lot of cloud providers who don't support externally available VIPs. They do, however, offer their own nearly-identical solution (such as Elastic IPs from Amazon).
The use of VIPs or similar could have potentially avoided the need for such a risky upgrade, potentially also saving millions of dollars in the process. Of course, I could simply be missing some hidden requirement from customers that they couldn't use VIPs but that's pretty uncommon, even in the finance industry.
Seems counter intuitive to run virtual appliances on static addresses if it can be avoided.
I wouldn't be proud of this, quite the opposite. I would suggest critically reviewing the entire infrastructure management strategy since months lost to a single upgrade is obviously indicative of greater problems.
Instead of just provisioning fresh VMs and migrating customer data they're doing this massive upgrade in place on existing machines to avoid losing the assigned IPs.
I guess they decided the benefits of being cloud provider agnostic outweighed the downside of spending months of man hours automating in-place OS upgrades.
I've worked on a Windows project where we solved a similar problem by booting from vhd, so you can "just" write a new vhd, uniquify it with the per machine config, update the boot menu and reboot - all data are on a separate volume, naturally
I'm surprised they went to all this effort for only 2000 machines though.
Again, very impressive as an academic exercise, especially considering the given script isn't actually that complicated, but wow, they had some serious guts running this in production!
That reminds me of this crazy video I once watched. https://youtube.com/watch?v=Cad8fyYeFnY
