undefined | Better HN

0 pointsgreedo5y ago0 comments

Sometimes. And sometimes they make the same mistake over and over.

We had an admin in charge of our storage. He had worked with our old vendor's SAN for years, then we got a new SAN. Trained him/certified him etc. He "accidentally" shut down the entire SAN. That brought down the entire company for over 9 hours.

Fast forward two years later, he screwed up again and caused a storage outage affecting about 1100 VMs. Luckily not much data loss, but a painful outage.

Then a month ago, he offlines part of the SAN.

Some people never learn, and recognizing this early is usually better than letting someone continue to risk things.

0 comments

mehrdadn5y ago

3 mistakes in... >2 years? I feel like it's really hard to tell if the problem is really the person at that point. Have you had others perform the same job for a similar duration to see if they avoid the same mistakes?

nitrogen5y ago

If you made a list of every mistake each person makes in 2-3 years, and omitted all other detail, pretty much everybody would look like a terrible person. Context, frequency, etc. all matter.

If particular systems or people are seeing a high frequency of mistakes, maybe the system design is at fault, not just the person. Obviously it's hard to do in practice, but the ideal is to design systems that are mistake proof.

greedoOP5y ago

This is just the mistakes made in the SAN/Storage part of his responsibilities. As we used to say in World of Warcraft, "Can't heal stupid."

jodrellblank5y ago

> "He had worked with our old vendor's SAN for years, then we got a new SAN."

Great way to invalidate years of experience. Presumably from your telling of the story, he didn't cause problems with the old vendor's SAN?

> "He "accidentally" shut down the entire SAN."

So, was it an accident, or was it an "accident"? You can't have it being a mistake if you're also hinting it was deliberate and malicious.

greedoOP5y ago

He was trained and certified on the new SAN, and surely some of his prior experience on the legacy SAN would translate. Just as moving from AIX to RHEL/CentOS wouldn't invalidate all your skills and experience.

It was a real accident when he shut down the SAN the first time. I don't know why I put it in scare quotes.

j / k navigate · click thread line to collapse

0 pointsgreedo5y ago0 comments

Sometimes. And sometimes they make the same mistake over and over.

Fast forward two years later, he screwed up again and caused a storage outage affecting about 1100 VMs. Luckily not much data loss, but a painful outage.

Then a month ago, he offlines part of the SAN.

Some people never learn, and recognizing this early is usually better than letting someone continue to risk things.

0 comments

mehrdadn5y ago

nitrogen5y ago

If you made a list of every mistake each person makes in 2-3 years, and omitted all other detail, pretty much everybody would look like a terrible person. Context, frequency, etc. all matter.

greedoOP5y ago

This is just the mistakes made in the SAN/Storage part of his responsibilities. As we used to say in World of Warcraft, "Can't heal stupid."

jodrellblank5y ago

> "He had worked with our old vendor's SAN for years, then we got a new SAN."

Great way to invalidate years of experience. Presumably from your telling of the story, he didn't cause problems with the old vendor's SAN?

> "He "accidentally" shut down the entire SAN."

So, was it an accident, or was it an "accident"? You can't have it being a mistake if you're also hinting it was deliberate and malicious.

greedoOP5y ago

It was a real accident when he shut down the SAN the first time. I don't know why I put it in scare quotes.

j / k navigate · click thread line to collapse