There was 250+ dedicated servers, 2-3 weeks of restoring week-old backups (thankfully they had these weekly intervals kept offline). Mass exodus of clients.
"Ex-employee" used root keys and a boot zerofill drop and rebooted every server resulting in severe data loss. Their online backup systems were also using these keys and we're not spared.
They said they would have to shut down the company as a result, but ended up securing capital and eventually launching what would become digitalocean.
They said it was highly probable that it was an ex employee and that the FBI was investigating buy nothing was released about it.
Good cautionary tale for segregation of credentials and proper user key management.
> In 2003, Ben and Moisey Uretsky who had founded ServerStack, a managed hosting business, wanted to create a new product which would combine the web hosting and virtual servers. The Uretskys, having surveyed the cloud hosting market felt that most hosting companies were targeting enterprise client leaving the entrepreneurial software developers market underserved. In 2011 the Uretskys founded DigitalOcean, a company which would provide server provisioning and cloud hosting for software developers.
Seems like the better option is keep your admins happy as much as possible.
Mario Savio was a Free Speach Activist and organized a protest to protect the Freedom of Speech at Berkeley around the 60s. In his speech to protestors, he says "there's a time when the operation of the machine becomes so odious... that you can't take part... and you've got to indicate to the people in charge that unless you're free, the machine will be prevented from running at all!" Applied to free speech, this notion of disrupting the functioning of an organization was lauded, because freedom of speech is just that important.
But let's shift to employment. Without employment, it's very hard to survive. And here's a situation where the people in charge has the upper hand in every arena- hiring, pay, work Place behavior... etc. How do we know that the ex-admin wasn't blackmailed by the CEO to come back to work for free to fix something, or future references will be negative? Why are we so quick to side with the employer in this matter when we know nothing of the situation at all? Why do we start calling the employee a felon? He hasn't even been charged yet.
My point is, context is important. Fine, corporations have the power to ruin your life as a deterrent to keep you from acting against their interests, and that's just the way society is. And fine, We're not all rational at every instance of life. The calculus of establishing status quo equilibrium of those two conditions/constraints is hard, but without context to the situation, who are we to decide who's right or wrong? Would you label Mario Savio wrong for protesting and urging protestors to prevent the operation of the college from functioning in the name of preserving Free speech wrong? No, because you've learned the context.
You can comfortably make a determination about what is right and wrong. We don't know the facts, but if the claims are true, wiping out not just that company's property, but that of their customers, is a crime.
Now, sometimes a crime is justified, but I don't think it is a rush to judgement to work from a starting point that criminal behaviour is bad until proven otherwise.
Why did this happen? How can we prevent it from happening in the future? These are the questions we need to stress.
In particular, why does an ex-employee still have access to production? I say when something like this happens and heads must roll, they must roll at the top. Fire the CEO. Fire the board. Leave the sysadmin alone.
This is a civil matter. My tax dollars should not pay for a criminal lawsuit. Screw that.
Oh and by the way if you're reading this: please help repeal cfaa.
I wish HNers would get their heads out of their tech arses and face the reality. Most people do _not_ have a choice where they work. Most people cannot afford to quit. Most people don't have jobs where they can increase their skillsets. Most people do not have time to read HN while they enjoy their 10 AM pause in their comfy sofa while working from home. You are, for the most part, amongst the most privileged people in the world. Yet you continue to spew the "employment is voluntary" propanganda because you never had to face actual hardships. Worse, you push people of lower classes even further in the ground, when what you should be doing is elevating them to make a better and fairer society.
Urbanization fundamentally has been engineered to create "wage slaves", who are basically modern sharecroppers (though the system has some extra steps).
It's called collateral damage. There is no context, outside of fantasy, where the admin could be in the right to do this.
If you treat people bad, they'll treat you bad. If you nuke Russia, Russia will nuke you. It's the Nash Equillibrium where each party is faced with a game and certain situations call for your best move. But your best move should account for what I'll do, and that should be factored into your initial move.
Mutually assured destruction is actually a powerful deterrent. Do we know what the ex-admin's situation was prior to all this? Only then should we pass judgement.
The collateral damage would low to non-existent for the users depending on how critical the service was. The direct damage would be to the employer. If the employer was the bad guy (hypothetically), would you still state there's no context or any situation whatsoever where the employee should cause them damage?
To us, it's not justifiable. Hell, the ex-admin may think so too. But that doesn't mean we should automatically side with the employer and subsequently crucify the ex-admin. We don't have any information yet.
Not free speech: punching your boss in the face, burning the building down, vandalizing the office, deleting all the data on the servers and hurting a bunch of people that had nothing to do with your conflict with management
I'm all about workers right and have walked out on many a job because management were raging assholes. I have convictions and I stand by them but conviction and doing the right thing in life often come with sacrifices. I'm currently underemployed because of my convictions but I can sleep at night knowing I'm doing the right thing. The way for us to take back power in the tech industry is to organize, unionize and refuse to work for abusive and exploitative employers. We're the ones with the skills. They need us not vise versa.
I don't know you.
But what I have learned that people who do not care about other peoples property, care very much about their own. Destroy other people business? Go for it! Destroy my car which I could kill someone with? No way, it's mine, I've worked hard for it. Vandalize houses of rich people? Go for it! Steal my iPhone? Hey, where is the police when you need them?
In Berlin people cheer the burning of other peoples BMWs - yes this is a thing. The same people go to court when the police scratches their table tennis table during a raid.
There is nothing to imply human rights have been violated. If such information is presented im sure people will respond in kind.
The most prominent example of this in recent memory is Peter Thiel sponsoring Hulk Hogan's lawsuit against Gawker. Do you approve of what he did?
Corporations function as de facto titles of nobility in the US, so crossing one is crossing social rank.
Lol. Citation please.
I'd like to see the justification for forcing thousands of customers into expensive disaster recovery because the company fired a sysadmin, likely for good reason.
BTW, we had a netadmin interview a few months ago. Guy was really smart, aced the technical and group interview. We were really looking forward to hiring him, and only needed to pass a background and reference check. HR told us in no uncertain terms to run the other way. They didn't share what was in his check but it wasn't good.
Always ask to see the background check details if you're the hiring manager (you should have the rights to see that despite what HR might tell you). Could be just a personal issue an HR employee might have with a former colleague. Or discrimination-based (happens).
If you're referring to private reference checks, of the type that would surface "personal issues people might have with formal colleagues", you're not entitled to anything whatsoever.
If you're a hiring manager in an organization where HR handles background checks, you personally as the hiring manager are entitled to nothing. I would venture further that it's inappropriate for HR to provide criminal/credit background check information to hiring managers.
I've had my own background checks in the past and no one has shared them with me. Speaking to HR, unless there is a negative item on your record they aren't required to.
As soon as individual managers want to make those decisions for themselves, it becomes much harder to refute claims of discrimination.
So, just because a guy has a bad reference it does not mean that he's necessarily bad.
This is so true. Years ago I was brought into a company specifically to improve their software quality process, but without being aware one of the company owners (of which there was two) was against be being employed for that purpose. I uncovered a lot of incompetence and outright corruption with some employees. The 'good' owner went on stress leave, and then pressure was brought to bear on me, resulting in me quitting on the spot one Monday morning; not my proudest moment, but I couldn't take the pressure any more, and my ally was no-where to be seen. Fast forward a decade, and I interviewed with someone who used to work at the same company; it turns out that my leaving had been framed as 'fired for incompetence', and word had been put around part of the local industry that I was hard to work with, unreliable, bad at my job etc. I laugh about it now, but at the time it really bothered me about the possible damage done to my career and reputation.
I don't know what country you're in so I won't speculate on what's legal or not in your area, but here in the U.S. it's illegal for a prior employer to provide false information to a prospective new employer during a background/reference check. If your former boss tried to blacklist you like that, he'd put his company at risk for a civil suit. And, while it's not illegal to truthfully say that a current or former employee is a bad employee, doing so rides the thin line of opening the company up to a libel lawsuit.
Generally speaking, a company might "dish the dirt" on a former employee if there are criminal charges to back up the claims. Even then, legal and HR will likely frown upon it. Usually, when the new company calls the old company for a reference, the old company will say something like "Yes, $employee worked here from $startDate to $endDate" and refuse to divulge any other information in order to avoid any semblance of libel.
Once again, this is my limited experience in SMB and government settings, I have no experience in mid-to-large businesses, Fortune 500 companies, and Silicon Valley startups. We've all heard stories about managers at such companies going around HR and discussing potential hires at the bar or on the golf course.
You'd be surprised. I did something somewhat similar and was convicted of a federal felony. No fast food place or retailer would touch me with that record. But ironically I've found plenty of IT work with smaller companies.
I always have a hard time with those contexts. HR sometimes has the wrong idea of what is unacceptable and what is appropriate. There really isn't a good reason for them not to tell you. The only thing that would make sense is if they shouldn't know it.
A similar thing happened to a client. Sysadmin logged into GCP and Azure immediately after termination and just deleted everything. He was in the UK, we were in the US. Wasn't worth it to try to get someone to prosecute, and I'm sure we're not listed as references.
Did get people motivated for a multifactor delete bucket for extra backups.
I was checking out the Sunday paper a few days later (this was the 90s), and it turned out that the guy was a fugitive who basically killed his wife and fed her to the fishes.
Pretty freaky stuff.
You flunked your own background check.
Asking previous employers about their experience with an employee, however is not illegal. It's usually not illegal for them to say something negative if it's true, though some businesses are conservative about what they will say out of fear of being sued for slander. Accessing public court records or news stories about criminal cases and using that information for employment purposes is usually not illegal. Asking prospective employees if they've been convicted of serious crimes is usually not illegal.
If a company walks away from this and doesn't take legal action; they should themselves get sued by their customers.
Destroying company property is a crime; wether it's defacing a website or ... it's not your property; you are just hired to maintain it (in one shape or another)
Regardless of what crime they may (or may not) have made; you are looking for the person and what they do. Not what they did.
This is a common problem; we look at the past a bias of the future. Life only works out that way if the person is too unwilling to change; and that again is something you should look for in the hiring process.
Lastly; hiring ex-hackers isn't a bad thing. Caring about a background check when hiring an ethical hacker or someone who turned their life around; only shames them and pushes them back where they came.
So be careful or you only end up criminalizing being a criminal.
Also, he is not a hacker, just an asshole. Having the admin login and password doesn't make you elite. The only weakness he exploited was himself.
Yeah, that. Also, secure backups and compartmentalized systems and data access.
https://web.archive.org/web/20170603212121/https://verelox.c...
I suspect that person will soon have a fair amount of time to decide on what new career to pursue to pay down the fines when they get out...
And having switched jobs quite a few times, the next one is always better for you, regardless.
It's not an equal relationship. One side usually has significantly more power than the other.
Apparently they did not.
At the end of the day, the people working at the company are the ones who are doing the work, and who have control of the means of production. The ex-admin's bosses probably thought they were the important ones, and that this worker was a replacable cog, but they found out the hard way that this was not the case.
I worked at a Fortune 100 investment bank where this happened. Everyone knew layoffs were coming. One week after layoffs came, a digital "bomb" went off wrecking many servers. So security went through, trying to find evidence (nothing incriminating from what I heard, although they had a strong suspect) and also looking for more bombs. They missed out on finding and defusing one, because another one went off a month later.
The view from the pinnacle, people counting the dividends on the checks that they inherited is that they're the job creators, and everyone else is dispensable. This company just found out that is not the case.
Therefore, to behave with integrity, you must have formulated your own set of values about what is "the right way" to behave.
Every minute of every day, we all have the option to behave with or without integrity in a whole range of ways.
You earn respect by demonstrating behaviours over time where you have taken the interests of others into consideration, generally people consider someone who behaves like this to have "integrity", especially when they continue to behave that way when no-one is looking.
Saying things like "The ex-admin's bosses probably thought they were the important ones" indicates a childish set of values where there is a power struggle between employers and employees ........ of course the "bosses" are the important ones, they act for the business which is an independent legal entity, upon which many people depend for their lives to work effectively. If, as an employee, you feel poorly treated or otherwise dissatisfied, then the right thing to do is leave in a polite and respectful manner, even if you feel you were not treated in that way. Depending on the circumstances, if you were actually treated really badly, then the right thing to do is pursue your complaint through the appropriate legal channels.
Someone important in my life once said to me "the only thing you have is your reputation". Take that reputation, defend it, enhance it, nurture it and earn the respect to grow it. Don't throw it in the garbage by smashing other people (or their business) in a childish tantrum. I admit this is hard to do - I regret many things I have done in my life, but I try to lead a life consistent with my own sets of values that I think are meaningful and I get rid of people from my life who I think don't have integrity, or whose values are different from mine in critically important ways.
I don't think this really indicates that at all. Maliciously inflicting damage on the company when you're fired is very different from being irreplaceable. It makes it risky to replace you, but that's not the same thing.
Sabotaging servers doesn't mean you're unreplacable any more than a terrorist attack means Western culture is depraved.
I'd believe you if the servers simply started falling apart without this person around, but that wasn't the case.
The solution to this, should it become a regular occurrence, is to make the folks with the keys to the kingdom replaceable.
It's doable, companies just don't do it because most people don't want to destroy their high paying and relatively comfortable careers committing felonies and getting sent to prison because they had to spend a few weeks or months looking for a new job.
As my own company is growing, we fully trust all employees, (limiting only what is essential), but, a dev ops guy if he was so inclined could technically do something like this... It always scares me.
For really important accounts - we have three people who each know two thirds of the password. It requires two people to then log in and do damage.
For example if the root password was CatDogFish then
Person1: CatDog_
Person2: _DogFish
Person3: Cat_Fish
Two people can then log in and watch what the other person is doing.
Because if not, once you are admin, you can install programs that let you become admin again at will.
But it's not a bad system.
Beyond that, be sure to keep regular backups (and test them), and audit all user actions. (feed the logs into something like Splunk, running on a separate machine)
And do backups. And then backups of those backups.
That's probably because when my grandmother died, my boss at QueBIT said "Ok, go home, call me when you can work again - however long that takes." There was never a discussion of PTO/HR policy, just human treatment.
Also remember to test restoring your backups or they don't count.
I worked retail to pay for college. Could always tell when a manager was getting the boot; they'd order new cylinders for all the doors. You kind of have to have that plan in place in IT too.
That's how I do it, anyway.
We can grant that this can be logistically difficult at certain scales, but it doesn't fall into the "engineering-impossible" bucket until you reach Facebook's size.
I don't know of any server provider (bare metal or cloud) that forces users to allow the company full access to their data (outside of managed providers were you voluntarily give this up , as you're paying them to fully manage your server)
If the CI/CD is done right, then no DevOps staff has any access to any servers and no one can delete anything except a scripts and AWS configurations.
The whole problem with limiting permissions is that you have to do all the work of deleting files, servers and drives.
covfefe
If someone is planning a malicious exit, it can be very hard to stop them depending on how "integrated" they are.
You can't. Not from an admin.
Same as how if you are rooted the only advice is to reinstall. It's simply impossible to reliably undo everything from inside the machine.
If you are a company, reimage the machine, then reinstall everything, and copy the code fresh from known good source control (and hope someone was watching source control that the admin did not check something in).
edit: also, use a bastion host which has the keys on it and don't allow them to be removed / used from laptops directly.
This problem is not as simple as you are pretending it is
Rephrase the question -- what idiot customer is going to do business with such a place that allows such a lapse in security to happen?
Intel would basically have to buy the company.
The kind of people that:
- Use Gmail, iCloud, etc. post Snowden
- Buys SSL certificates from Comodo, etc.
- [put other companies here]
What, exactly can be done to secure a company against a malicious systems admin? These are the guys typically with not only the keys to everything but also the knowledge of how it all works.
You say that the company cannot be trusted for "allowing" this to happen.
I know quite alot about this stuff, and for MOST companies, they simply have to trust that the people with the keys to the castle with behave responsibly.
There are ways to design infrastructure such that it is protected from its builders and keepers, but this is very very hard and complex and expensive.
Presumably you work for a company that has taken steps to ensure this will never happen, what are they?
Seriously though, would Verelox still be running unpatched AMT many weeks after the disclosure of this authentication bug? Or does GP think there are more bugs which Intel hopes to sweep under the rug forever by individually covering each incident? They would spend quite a money on these bribes while AMT bugs can simply be fixed with BIOS updates.
This Intel conspiracy doesn't make sense. It's aliens, folks, I know it.
(Btw, IMO there is no excuse or justification for any admin or exadmin to ever do this. Among many other issues is the fact he deleted the data/work of individuals who had nothing to do with whatever "problem" he has with Verelox )
There's probably excuses and justifications. I personally wouldn't do it and they're probably wrong for doing it but I don't want to jump to conclusions and moral absolutes so easily.
Nothing is foolproof, but anytime you've got constant network access to every last copy of your data, you're begging to lose it. It's the reason why people who think one copy (redundantly dispersed or not) in AWS S3 is sufficient scares me to death. Is it unlikely Amazon would get hacked and have the entire thing blown up? Sure... but if we go to war with China I wouldn't want to bet my company on it.
I know you meant it as an example, but this sort of extreme attitude towards security is just another footgun.
Probably not in the immediate aftermath, but someone might decades later, if the company actually does something valuable.
Why? We went to war with Europe and Asia a few times and businesses kept chugging along here in the states.
What if the admin was a remote worker in a country that doesn't have an extradition treaty with the Netherlands (Verelox hq country)?
Maybe in North Korea, or the US. Relatively unlikely in civilized countries.
Possibly related?
Edit: Make that the 7th based on: https://www.facebook.com/Verelox/posts/1886196381643427?comm...
Or these two events are unrelated. Or the whole deleted prod on day 1 story is made up.
I'd agree that defending against malicious admins is really difficult. We have really little context to go by here, but I think there is important distinction to be made if the malicious actions (planting backdoors or whatnot) were done while the malicious actor was still employed or after their employment was terminated. Proper exit procedures protect against the latter, but generally are not that effective against the former.
> but in the end humans make mistakes
And it is useful for us outsiders to highlight the real mistakes so that we can learn from them, because that is really the biggest value of stories like this for the majority of people who are not directly impacted.
Note that I don't know the details and am making assumptions that may be wrong about the case in question, but in general, if you can't deny access quickly to any given account, you really want to fix that. Not just because of rogue ex-employees - what happens when $important_person's account is compromised?
Some posts from Verelox staff towards bottom third of this forum page search for user name Verelox
Otherwise while the vast majority of your staff will be decent people and not cause problems like this, it just takes one angry ex staff member with a grudge to cause problems.
They also need to revise their backup system too. There should rarely if ever be a risk that any data is 'unrecoverable', yet their update says some data will just be impossible to get back.
As for the employee involved... well I hope they like the inevitable lawsuit their selfish, stupid actions will bring them. I don't care what you think of a company you worked for, there's no excuse to destroy their business through actions like this. Also, good luck getting any jobs in the industry after too. Because with this on your track record, no one will touch you with a ten foot bargepole.
So yeah, what a disaster all round.
I'd like to know more, I think...
Are you sure no matter what the company did? What if the CEO threatened the ex-admins family? Or if the ex-admin found child porn on the CEOs computer?
There's a fine line between right and wrong in most situations. The most egregious acts of disobedience can be seen as defiance or foolish. It's not for you to decide- especially when there isn't any context to this whole situation.