Mario Savio was a Free Speach Activist and organized a protest to protect the Freedom of Speech at Berkeley around the 60s. In his speech to protestors, he says "there's a time when the operation of the machine becomes so odious... that you can't take part... and you've got to indicate to the people in charge that unless you're free, the machine will be prevented from running at all!" Applied to free speech, this notion of disrupting the functioning of an organization was lauded, because freedom of speech is just that important.
But let's shift to employment. Without employment, it's very hard to survive. And here's a situation where the people in charge has the upper hand in every arena- hiring, pay, work Place behavior... etc. How do we know that the ex-admin wasn't blackmailed by the CEO to come back to work for free to fix something, or future references will be negative? Why are we so quick to side with the employer in this matter when we know nothing of the situation at all? Why do we start calling the employee a felon? He hasn't even been charged yet.
My point is, context is important. Fine, corporations have the power to ruin your life as a deterrent to keep you from acting against their interests, and that's just the way society is. And fine, We're not all rational at every instance of life. The calculus of establishing status quo equilibrium of those two conditions/constraints is hard, but without context to the situation, who are we to decide who's right or wrong? Would you label Mario Savio wrong for protesting and urging protestors to prevent the operation of the college from functioning in the name of preserving Free speech wrong? No, because you've learned the context.
You can comfortably make a determination about what is right and wrong. We don't know the facts, but if the claims are true, wiping out not just that company's property, but that of their customers, is a crime.
Now, sometimes a crime is justified, but I don't think it is a rush to judgement to work from a starting point that criminal behaviour is bad until proven otherwise.
Why did this happen? How can we prevent it from happening in the future? These are the questions we need to stress.
In particular, why does an ex-employee still have access to production? I say when something like this happens and heads must roll, they must roll at the top. Fire the CEO. Fire the board. Leave the sysadmin alone.
This is a civil matter. My tax dollars should not pay for a criminal lawsuit. Screw that.
Oh and by the way if you're reading this: please help repeal cfaa.
Put another way, someone at Verelox screwed up and left the door unlocked, but that doesn't mean that the person who walked in broke stuff is in the clear.
Look, someone, without authorization, accessed a former employer's network and maliciously destroyed data. That's a crime. Sure, the CFAA is overly broad and is abused, but this is not one of those cases: this is a textbook example of something that should be prosecuted under the CFAA.
We may remain open to additional information without presuming that uncivil and illegal vandalism was justified indeed without inventing a narrative from whole cloth as you have done. The logical conclusion is that drawing from your own life experience you identify so strongly with the narrative of the wronged sysadmin that you desire to fit a narrative to sparse facts that has no basis in fact.
We are merely commenting on a story on hacker news. We aren't members of the jury and don't face the same burden or power. I'm down with repealing the cfaa because its badly written, I'm down with figuring out who dropped the ball as far as giving the sysadmin access post firing, but as to the sysadmin himself, burn the witch!
IIRC, the Soviet Union had a policy rather like this, referred to as something like the Vertical Stroke, where anytime there was a screw-up at a low level, they would fire the screw-up-ee's manager, and manager's manager, and so on, up to a very high level. The practical result was a drastic decrease in innovation and risk-taking. CEOs and others at that level usually aren't close enough to the guys actually doing direct work to supervise them all closely enough to ensure they don't make mistakes. All they can do is create a culture where there's a book of rules, and you don't deviate from the rules ever, for any reason, no matter what. So that's what they do, and that's the resulting culture and economy that you get.
Maybe we shouldn't rush to judge either sysadmins or CEOs, but instead figure out who, if anyone, actually did something malicious, and let everyone else take the lessons they've already learned from what happened.
If he did it (intentionally wiping the servers), then he is a criminal and a bad person.
If he is a scapegoat, then he didn't do it, so that's a totally different situation.
I don't see how you can disagree with either of these statements.
And discussion of how he still had access is an unrelated matter.
Maybe he became an ex-employee after (and as a result of) wiping production.
So if an aggrieved ex employee does enough damage that the company has insufficient resources to sue, or has enough resources themselves to make that difficult, it's all good?
Individuals and companies should not be vulnerable to attacks like this based on their resources. It's in all our interests to ensure this sort if activity is dealt with severely because we are all vulnerable. Mutual defence in the form of criminal prosecution of offenders is the way they go IMHO.
How many little shops rely on Verelox - aren't they little guys?
How many end customers depend on services that depend on Verelox - you know someone like you - aren't they the little guys?
The police/prosecuting agency in the country where the company is based I imagine. That appears to be Holland [1]
[1] https://www.ripe.net/membership/indices/data/us.verelox.html
The fact that it was an ex-administrator does suggest foul play as opposed to a mistake compounded with poor backup processes. We shall have to see how the story pans out.
Sigh. I can't believe we're going there on this one. Just so we're clear, you think that maybe the definition of criminal behaviour shouldn't include wiping out systems without the owner's consent?
> And why is criminal behaviour always placed on the little guy when companies can always get the leeway of claiming ignorance?
I don't know what you are talking about. Criminal behaviour is placed on the criminal... and ignorance of the law is not a defense.
I wish HNers would get their heads out of their tech arses and face the reality. Most people do _not_ have a choice where they work. Most people cannot afford to quit. Most people don't have jobs where they can increase their skillsets. Most people do not have time to read HN while they enjoy their 10 AM pause in their comfy sofa while working from home. You are, for the most part, amongst the most privileged people in the world. Yet you continue to spew the "employment is voluntary" propanganda because you never had to face actual hardships. Worse, you push people of lower classes even further in the ground, when what you should be doing is elevating them to make a better and fairer society.
Urbanization fundamentally has been engineered to create "wage slaves", who are basically modern sharecroppers (though the system has some extra steps).
Suppose someone was born out in the forest, and there were no cities, no societies, no technology. Would you call that person a sustenance slave? Would you say that their foraging was involuntary?
Being employed at a particular organization is, in fact, voluntary. You are not obligated to stay at one employer forever.
No one has the perfect job, everyone thinks they should make more, have more benefits, more freedom. But everyone has choices. You don't have to participate in the rat race, you can stop letting material possessions drive your lifestyle decisions.
It's called collateral damage. There is no context, outside of fantasy, where the admin could be in the right to do this.
If you treat people bad, they'll treat you bad. If you nuke Russia, Russia will nuke you. It's the Nash Equillibrium where each party is faced with a game and certain situations call for your best move. But your best move should account for what I'll do, and that should be factored into your initial move.
Mutually assured destruction is actually a powerful deterrent. Do we know what the ex-admin's situation was prior to all this? Only then should we pass judgement.
You are basically justifying the 'killing of innocents' here..
The collateral damage would low to non-existent for the users depending on how critical the service was. The direct damage would be to the employer. If the employer was the bad guy (hypothetically), would you still state there's no context or any situation whatsoever where the employee should cause them damage?
To us, it's not justifiable. Hell, the ex-admin may think so too. But that doesn't mean we should automatically side with the employer and subsequently crucify the ex-admin. We don't have any information yet.
Not free speech: punching your boss in the face, burning the building down, vandalizing the office, deleting all the data on the servers and hurting a bunch of people that had nothing to do with your conflict with management
I'm all about workers right and have walked out on many a job because management were raging assholes. I have convictions and I stand by them but conviction and doing the right thing in life often come with sacrifices. I'm currently underemployed because of my convictions but I can sleep at night knowing I'm doing the right thing. The way for us to take back power in the tech industry is to organize, unionize and refuse to work for abusive and exploitative employers. We're the ones with the skills. They need us not vise versa.
I don't know you.
But what I have learned that people who do not care about other peoples property, care very much about their own. Destroy other people business? Go for it! Destroy my car which I could kill someone with? No way, it's mine, I've worked hard for it. Vandalize houses of rich people? Go for it! Steal my iPhone? Hey, where is the police when you need them?
In Berlin people cheer the burning of other peoples BMWs - yes this is a thing. The same people go to court when the police scratches their table tennis table during a raid.
There is nothing to imply human rights have been violated. If such information is presented im sure people will respond in kind.
The most prominent example of this in recent memory is Peter Thiel sponsoring Hulk Hogan's lawsuit against Gawker. Do you approve of what he did?
Corporations function as de facto titles of nobility in the US, so crossing one is crossing social rank.
I think that's very on point. While HN is a technology site, it's also one for "founders" * who aspire join the elite of that social order, which explains the exaggerated empathy for the interests of shareholders and companies.
* (and those with fantasies of being one)
Lol. Citation please.
I'd like to see the justification for forcing thousands of customers into expensive disaster recovery because the company fired a sysadmin, likely for good reason.
BTW, we had a netadmin interview a few months ago. Guy was really smart, aced the technical and group interview. We were really looking forward to hiring him, and only needed to pass a background and reference check. HR told us in no uncertain terms to run the other way. They didn't share what was in his check but it wasn't good.
Always ask to see the background check details if you're the hiring manager (you should have the rights to see that despite what HR might tell you). Could be just a personal issue an HR employee might have with a former colleague. Or discrimination-based (happens).
One day one of the feared internal security team who had caught him years ago was in the building and met him and freaked out - our centre mangers politely told him to f off when he demanded that we fire the guy.
If you're referring to private reference checks, of the type that would surface "personal issues people might have with formal colleagues", you're not entitled to anything whatsoever.
If you're a hiring manager in an organization where HR handles background checks, you personally as the hiring manager are entitled to nothing. I would venture further that it's inappropriate for HR to provide criminal/credit background check information to hiring managers.
I've had my own background checks in the past and no one has shared them with me. Speaking to HR, unless there is a negative item on your record they aren't required to.
As soon as individual managers want to make those decisions for themselves, it becomes much harder to refute claims of discrimination.
"Your honor, my client, the recently released child rapist with an iffy work record, is suing because BigCorp says they normally don't hire felons, but a few years ago BigCorp made an exception for a felon who sold marijuana, got his college degree in prison, then after he was released worked at non-profits helping the poor while getting an advanced degree in computer technology and writing several highly regarded academic papers on improving user interfaces for the disabled. That's not fair!"
The specific facts matter, and HR should be fired if they have blanket polices that ignore them. HR needs to serve the business, not the other way round.
So, just because a guy has a bad reference it does not mean that he's necessarily bad.
This is so true. Years ago I was brought into a company specifically to improve their software quality process, but without being aware one of the company owners (of which there was two) was against be being employed for that purpose. I uncovered a lot of incompetence and outright corruption with some employees. The 'good' owner went on stress leave, and then pressure was brought to bear on me, resulting in me quitting on the spot one Monday morning; not my proudest moment, but I couldn't take the pressure any more, and my ally was no-where to be seen. Fast forward a decade, and I interviewed with someone who used to work at the same company; it turns out that my leaving had been framed as 'fired for incompetence', and word had been put around part of the local industry that I was hard to work with, unreliable, bad at my job etc. I laugh about it now, but at the time it really bothered me about the possible damage done to my career and reputation.
I don't know what country you're in so I won't speculate on what's legal or not in your area, but here in the U.S. it's illegal for a prior employer to provide false information to a prospective new employer during a background/reference check. If your former boss tried to blacklist you like that, he'd put his company at risk for a civil suit. And, while it's not illegal to truthfully say that a current or former employee is a bad employee, doing so rides the thin line of opening the company up to a libel lawsuit.
Generally speaking, a company might "dish the dirt" on a former employee if there are criminal charges to back up the claims. Even then, legal and HR will likely frown upon it. Usually, when the new company calls the old company for a reference, the old company will say something like "Yes, $employee worked here from $startDate to $endDate" and refuse to divulge any other information in order to avoid any semblance of libel.
Once again, this is my limited experience in SMB and government settings, I have no experience in mid-to-large businesses, Fortune 500 companies, and Silicon Valley startups. We've all heard stories about managers at such companies going around HR and discussing potential hires at the bar or on the golf course.
You'd be surprised. I did something somewhat similar and was convicted of a federal felony. No fast food place or retailer would touch me with that record. But ironically I've found plenty of IT work with smaller companies.
I always have a hard time with those contexts. HR sometimes has the wrong idea of what is unacceptable and what is appropriate. There really isn't a good reason for them not to tell you. The only thing that would make sense is if they shouldn't know it.
A similar thing happened to a client. Sysadmin logged into GCP and Azure immediately after termination and just deleted everything. He was in the UK, we were in the US. Wasn't worth it to try to get someone to prosecute, and I'm sure we're not listed as references.
Did get people motivated for a multifactor delete bucket for extra backups.
I was checking out the Sunday paper a few days later (this was the 90s), and it turned out that the guy was a fugitive who basically killed his wife and fed her to the fishes.
Pretty freaky stuff.
You flunked your own background check.
Asking previous employers about their experience with an employee, however is not illegal. It's usually not illegal for them to say something negative if it's true, though some businesses are conservative about what they will say out of fear of being sued for slander. Accessing public court records or news stories about criminal cases and using that information for employment purposes is usually not illegal. Asking prospective employees if they've been convicted of serious crimes is usually not illegal.
If a company walks away from this and doesn't take legal action; they should themselves get sued by their customers.
Destroying company property is a crime; wether it's defacing a website or ... it's not your property; you are just hired to maintain it (in one shape or another)
Regardless of what crime they may (or may not) have made; you are looking for the person and what they do. Not what they did.
This is a common problem; we look at the past a bias of the future. Life only works out that way if the person is too unwilling to change; and that again is something you should look for in the hiring process.
Lastly; hiring ex-hackers isn't a bad thing. Caring about a background check when hiring an ethical hacker or someone who turned their life around; only shames them and pushes them back where they came.
So be careful or you only end up criminalizing being a criminal.
Also, he is not a hacker, just an asshole. Having the admin login and password doesn't make you elite. The only weakness he exploited was himself.
Yeah, that. Also, secure backups and compartmentalized systems and data access.
As my own company is growing, we fully trust all employees, (limiting only what is essential), but, a dev ops guy if he was so inclined could technically do something like this... It always scares me.
For really important accounts - we have three people who each know two thirds of the password. It requires two people to then log in and do damage.
For example if the root password was CatDogFish then
Person1: CatDog_
Person2: _DogFish
Person3: Cat_Fish
Two people can then log in and watch what the other person is doing.
Because if not, once you are admin, you can install programs that let you become admin again at will.
But it's not a bad system.
Beyond that, be sure to keep regular backups (and test them), and audit all user actions. (feed the logs into something like Splunk, running on a separate machine)
And do backups. And then backups of those backups.
That's probably because when my grandmother died, my boss at QueBIT said "Ok, go home, call me when you can work again - however long that takes." There was never a discussion of PTO/HR policy, just human treatment.
Also remember to test restoring your backups or they don't count.
I worked retail to pay for college. Could always tell when a manager was getting the boot; they'd order new cylinders for all the doors. You kind of have to have that plan in place in IT too.
That's how I do it, anyway.
We can grant that this can be logistically difficult at certain scales, but it doesn't fall into the "engineering-impossible" bucket until you reach Facebook's size.
I don't know of any server provider (bare metal or cloud) that forces users to allow the company full access to their data (outside of managed providers were you voluntarily give this up , as you're paying them to fully manage your server)
If the CI/CD is done right, then no DevOps staff has any access to any servers and no one can delete anything except a scripts and AWS configurations.
The whole problem with limiting permissions is that you have to do all the work of deleting files, servers and drives.
covfefe
And having switched jobs quite a few times, the next one is always better for you, regardless.
It's not an equal relationship. One side usually has significantly more power than the other.
* The irrational biases of other companies (even if we didn't care about you being a university grad, our commensurately lower wage offering reflected the fact that other companies did).
* If the employee telegraphed an air of needing the job (e.g. people with a debts), they got offered commensurately less.
It's an ugly process, truth be told.
Apparently they did not.
At the end of the day, the people working at the company are the ones who are doing the work, and who have control of the means of production. The ex-admin's bosses probably thought they were the important ones, and that this worker was a replacable cog, but they found out the hard way that this was not the case.
I worked at a Fortune 100 investment bank where this happened. Everyone knew layoffs were coming. One week after layoffs came, a digital "bomb" went off wrecking many servers. So security went through, trying to find evidence (nothing incriminating from what I heard, although they had a strong suspect) and also looking for more bombs. They missed out on finding and defusing one, because another one went off a month later.
The view from the pinnacle, people counting the dividends on the checks that they inherited is that they're the job creators, and everyone else is dispensable. This company just found out that is not the case.
Therefore, to behave with integrity, you must have formulated your own set of values about what is "the right way" to behave.
Every minute of every day, we all have the option to behave with or without integrity in a whole range of ways.
You earn respect by demonstrating behaviours over time where you have taken the interests of others into consideration, generally people consider someone who behaves like this to have "integrity", especially when they continue to behave that way when no-one is looking.
Saying things like "The ex-admin's bosses probably thought they were the important ones" indicates a childish set of values where there is a power struggle between employers and employees ........ of course the "bosses" are the important ones, they act for the business which is an independent legal entity, upon which many people depend for their lives to work effectively. If, as an employee, you feel poorly treated or otherwise dissatisfied, then the right thing to do is leave in a polite and respectful manner, even if you feel you were not treated in that way. Depending on the circumstances, if you were actually treated really badly, then the right thing to do is pursue your complaint through the appropriate legal channels.
Someone important in my life once said to me "the only thing you have is your reputation". Take that reputation, defend it, enhance it, nurture it and earn the respect to grow it. Don't throw it in the garbage by smashing other people (or their business) in a childish tantrum. I admit this is hard to do - I regret many things I have done in my life, but I try to lead a life consistent with my own sets of values that I think are meaningful and I get rid of people from my life who I think don't have integrity, or whose values are different from mine in critically important ways.
I don't think this really indicates that at all. Maliciously inflicting damage on the company when you're fired is very different from being irreplaceable. It makes it risky to replace you, but that's not the same thing.
Sabotaging servers doesn't mean you're unreplacable any more than a terrorist attack means Western culture is depraved.
I'd believe you if the servers simply started falling apart without this person around, but that wasn't the case.
The solution to this, should it become a regular occurrence, is to make the folks with the keys to the kingdom replaceable.
It's doable, companies just don't do it because most people don't want to destroy their high paying and relatively comfortable careers committing felonies and getting sent to prison because they had to spend a few weeks or months looking for a new job.
If someone is planning a malicious exit, it can be very hard to stop them depending on how "integrated" they are.
You can't. Not from an admin.
Same as how if you are rooted the only advice is to reinstall. It's simply impossible to reliably undo everything from inside the machine.
If you are a company, reimage the machine, then reinstall everything, and copy the code fresh from known good source control (and hope someone was watching source control that the admin did not check something in).
edit: also, use a bastion host which has the keys on it and don't allow them to be removed / used from laptops directly.
This problem is not as simple as you are pretending it is
If there isn't a police report, and charges aren't pressed against a malicious individual, then the company may be at fault.
Certainly the employee could willingly and pro-actively step down, out of personal guilt and feelings of shame, but then, one need not qualify with "ex" as simply "employee" will suffice. A criminal incident is worthy of such a clarification, indicating that the incident is a deliberate attack, but human error is not.
What if the admin was a remote worker in a country that doesn't have an extradition treaty with the Netherlands (Verelox hq country)?
If this was a "cyber crime" they could possibly be picked up to stand trial in any cooperating country which would narrow their choices of travel.
However, in that situation they would probably be fine except potentially limited ability to work for Companies who do background checks
> 4 low-latency locations in ISO/IEC 27001
> certified Tier 3+ data centers in United
> States, Netherlands, France and Canada.
Although that's perhaps not the right word. There is contiguous land between e.g. The Netherlands and Russia, so there's no "shore" involved as there would be between e.g. the USA and Russia. I do think the word's meaning is now more "another country", rather than denoting crossing physical land/water boundaries.
Maybe in North Korea, or the US. Relatively unlikely in civilized countries.
Rephrase the question -- what idiot customer is going to do business with such a place that allows such a lapse in security to happen?
Intel would basically have to buy the company.
The kind of people that:
- Use Gmail, iCloud, etc. post Snowden
- Buys SSL certificates from Comodo, etc.
- [put other companies here]
What, exactly can be done to secure a company against a malicious systems admin? These are the guys typically with not only the keys to everything but also the knowledge of how it all works.
You say that the company cannot be trusted for "allowing" this to happen.
I know quite alot about this stuff, and for MOST companies, they simply have to trust that the people with the keys to the castle with behave responsibly.
There are ways to design infrastructure such that it is protected from its builders and keepers, but this is very very hard and complex and expensive.
Presumably you work for a company that has taken steps to ensure this will never happen, what are they?
Seriously though, would Verelox still be running unpatched AMT many weeks after the disclosure of this authentication bug? Or does GP think there are more bugs which Intel hopes to sweep under the rug forever by individually covering each incident? They would spend quite a money on these bribes while AMT bugs can simply be fixed with BIOS updates.
This Intel conspiracy doesn't make sense. It's aliens, folks, I know it.
I'd agree that defending against malicious admins is really difficult. We have really little context to go by here, but I think there is important distinction to be made if the malicious actions (planting backdoors or whatnot) were done while the malicious actor was still employed or after their employment was terminated. Proper exit procedures protect against the latter, but generally are not that effective against the former.
> but in the end humans make mistakes
And it is useful for us outsiders to highlight the real mistakes so that we can learn from them, because that is really the biggest value of stories like this for the majority of people who are not directly impacted.
Note that I don't know the details and am making assumptions that may be wrong about the case in question, but in general, if you can't deny access quickly to any given account, you really want to fix that. Not just because of rogue ex-employees - what happens when $important_person's account is compromised?
There was 250+ dedicated servers, 2-3 weeks of restoring week-old backups (thankfully they had these weekly intervals kept offline). Mass exodus of clients.
"Ex-employee" used root keys and a boot zerofill drop and rebooted every server resulting in severe data loss. Their online backup systems were also using these keys and we're not spared.
They said they would have to shut down the company as a result, but ended up securing capital and eventually launching what would become digitalocean.
They said it was highly probable that it was an ex employee and that the FBI was investigating buy nothing was released about it.
Good cautionary tale for segregation of credentials and proper user key management.
> In 2003, Ben and Moisey Uretsky who had founded ServerStack, a managed hosting business, wanted to create a new product which would combine the web hosting and virtual servers. The Uretskys, having surveyed the cloud hosting market felt that most hosting companies were targeting enterprise client leaving the entrepreneurial software developers market underserved. In 2011 the Uretskys founded DigitalOcean, a company which would provide server provisioning and cloud hosting for software developers.
Seems like the better option is keep your admins happy as much as possible.
If that's the only logic that successfully gets through to the boss, it's good logic.
Not necessarily causality, but I'd say there's at least a correlation, and a good enough reason to make office life as bearable as possible.
Nothing is foolproof, but anytime you've got constant network access to every last copy of your data, you're begging to lose it. It's the reason why people who think one copy (redundantly dispersed or not) in AWS S3 is sufficient scares me to death. Is it unlikely Amazon would get hacked and have the entire thing blown up? Sure... but if we go to war with China I wouldn't want to bet my company on it.
I know you meant it as an example, but this sort of extreme attitude towards security is just another footgun.
Probably not in the immediate aftermath, but someone might decades later, if the company actually does something valuable.
Why? We went to war with Europe and Asia a few times and businesses kept chugging along here in the states.
https://web.archive.org/web/20170603212121/https://verelox.c...
I suspect that person will soon have a fair amount of time to decide on what new career to pursue to pay down the fines when they get out...
I'd like to know more, I think...
Are you sure no matter what the company did? What if the CEO threatened the ex-admins family? Or if the ex-admin found child porn on the CEOs computer?
There's a fine line between right and wrong in most situations. The most egregious acts of disobedience can be seen as defiance or foolish. It's not for you to decide- especially when there isn't any context to this whole situation.
Possibly related?
Edit: Make that the 7th based on: https://www.facebook.com/Verelox/posts/1886196381643427?comm...
Or these two events are unrelated. Or the whole deleted prod on day 1 story is made up.
(Btw, IMO there is no excuse or justification for any admin or exadmin to ever do this. Among many other issues is the fact he deleted the data/work of individuals who had nothing to do with whatever "problem" he has with Verelox )
There's probably excuses and justifications. I personally wouldn't do it and they're probably wrong for doing it but I don't want to jump to conclusions and moral absolutes so easily.
Some posts from Verelox staff towards bottom third of this forum page search for user name Verelox
Otherwise while the vast majority of your staff will be decent people and not cause problems like this, it just takes one angry ex staff member with a grudge to cause problems.
They also need to revise their backup system too. There should rarely if ever be a risk that any data is 'unrecoverable', yet their update says some data will just be impossible to get back.
As for the employee involved... well I hope they like the inevitable lawsuit their selfish, stupid actions will bring them. I don't care what you think of a company you worked for, there's no excuse to destroy their business through actions like this. Also, good luck getting any jobs in the industry after too. Because with this on your track record, no one will touch you with a ten foot bargepole.
So yeah, what a disaster all round.