A vulnerability on this scale.. and they pay out only 40K ? I don't understand, this is peanuts compared to the damages this could cause no? Why do they not pay out respectable amounts?
Should a surgeon ask you for millions? After all he saved your life.
What about the mechanic that finds an issue with your breaks?
If you paid someone not for the work done, but for the resulting loss if criminals exploited something, a lot of professions would ask for crazy prices.
If there are 50 others who can fix my brakes, one of them will likely try to undercut your price so as to get my business.
In this case, M$ could have paid ten times that amount and it would still be pocket money both in terms of their ledger AND vs the damages the PR could cause alone.
M$ does have black market competitors that would have paid far more.
Doing the right thing morally is good. But when people see time and again that simply cashing out once could set you up for life.(Maybe not in this case but maybe in others.) That is an awfully tempting risk to take and I wouldn't blame them.
We're literally saying it doesn't pay to be moral.
How have you confirmed that this amount is accurate? Have there been bo zero-days on the black market, instead all hackers have gone to microsoft?
Would the number of hack on black market change is amount paid out increased 10x? It seems you have no basis for claiming that the amount paid is optimal
But if no one can succesfully sue for those damages, then what is it really worth.
The idea that Microsoft with its gargantuan resources cannot discover these mistakes and has to wait for "security researchers" to discover them and then pay them a paltry sum for their "valuable work" makes little sense; a more sensible interpretation is that the company has little incentive to find such mistakes because it has little exposure to potential liability arising from them. It needs to stage "security theater" to avoid reputational damage but does not need to achieve real results. Mistake after mistake, the stock price is not affected. Regarding an ongoing lapse in quality control that spans four decades, it has no skin in the game.
I'd be in favor of something like this. Why should the criminal justice system and every tax payer be on the hook for protecting these big companies from the consequences of their bug-ridden software?
To me it seemed vaguely interesting but a little bit overpriced for real-world usage outside of fortune 500s or big government, where budgets aren't always a concern.
For comparison, I had the same eventhub feed Kafka Connect to dump it into Mongo 5.0's new timestream collections on a $200/month VM, and after running both outputs at full volume for a couple days, Azure offered me a helpful suggestion to downsize the under-utilized VM and save money.
I haven't quite figured out if the RU rate is just too high to be economical, and counts on large orgs with large data to expect a large bill; or if the RU model simple works badly for high velocity data. I've verified that the pricing model now scales down properly for small/medium loads, at which point the various high availability/geographcial distribution features really add value. I'm trying to conceptualize how the RU cost, amortized across longer term storage and less demanding ingress/egress schedules, can work out. For our scenario, Cosmos functionally offers exactly the horizontal scalability we need.
As a different sanity check, I set up a SQL Server account as the ingestion database and saw comparable costs, so it wasn't that Cosmos itself has an unworkable pricing model.
I've recently tested it as needed geospartial query support for a hobby project and the cost is cents per month with the serverless pricing model. I'm also accessing CosmosDB outside of Azure to use low cost VPS hosting and latency is pretty acceptable for web type stuff.
If you are DB read/write heavy obviously pricing is going to be higher due to pay for what you use, pre-provisioned pricing will become the better value option at that point.
I couldn't justify it prior to the serverless introduction; but I'm doing 1m+ ops/mo now for <1usd for some high-concurrency state tracking ops.
It has a MongoDB API Layer and a Cassandra API Layer (as well as SQL which is preferred).
Works just fine, apparently cost can blow out though with high usage, I haven't got there yet.
Does have a free tier.
You must be diligent to develop your application to use it efficiently from both a cost and performance/latency perspective.
Those that put in the work may not love it, but they are very successful at operating at scale. Massive, insane scale.
We use it with a private endpoint and turn off public access. I've been generally happy with it.
As long you have a good partitioning strategy seems work without much effort put into it.
> So you can imagine our surprise when we were able to gain complete unrestricted access to the accounts and databases of several thousand Microsoft Azure customers, including many Fortune 500 companies.
This is basically worst-case scenario for a database provider.
https://www.wiz.io/blog/chaosdb-how-we-hacked-thousands-of-a...
Microsoft gets a temporary mark on their reputation until their pr and marketing departments make us forget about it. That’s it. Nobody is even going to lose their bonus over this.
Seems like their resources (billions in cash) aren't allocated correctly.
Incompetence. It’s not the exclusive domain of small companies. I’ve dealt with some very incompetent people at Fortune 500 companies. As a matter of fact, the very people responsible for the mishap we are discussing now work at a Fortune 500 company.
...but due to a poor deployment strategy, the instances were not sandboxed from each other.
(and yes, you would be correct in thinking that was a catastrophic failure; it's like having a folder with a directory per customer and a master access key in each folder, and WOOOPS, didn't realize people might look at ../Amazon/master_key.txt
It's pretty hard to brush this one off; it's a colossal screw up)
I’ve seen some azure documentation that stresses you should never ever use your primary keys in a deployed service. Yet they do it willy nilly.
You see the former with a lot of “online Python interpreter” sites.
1. My service's users would have to grant my service permission to read all access keys for the queue
2. My service would look through the access keys and try to find one matching some conditions (specific provided name and read-only)
3. If it couldn't find a limited one (which it would not in pretty much every real case due to the defaults) it would grab any read/write key and use that
(Later, we used Managed Identity when that became available, which avoids the song-and-dance but also has performance issues with very large scale sets. Out of the frying pan...)
I'm wondering if Jupyter had similar ability to read the primary r/w key for every database that had it enabled.
There's a vulnerability in some jupyter notebook implementation allowing them access to unrelated notebooks which hold read/write keys to database they're connected to.
On one hand, my understanding is that they used TLA+ to validate the model[0]. One would assume that if they went to that trouble there is at least a specification for how it should work.
OTOH, Specifications can be flawed. This security issue appears unrelated to the design of Cosmos DB itself tho.
This bug is a security issues in an entirely separate component. No formal logic is going to warn you about exposed read/write keys.
Notably, some of the customers mentioned in the security researcher's blog are the type that have exploding wallets.
As in: "My wallet is bursting open because there's too much money in it! Mr Cloud Sales Guy, can you help me with this problem?"
if this is amazing I'd hate to see bad
It took a few hours for Microsoft to fix the problem, they responded within a half hour or so of the first complaint.
Two days seems like a long time for Microsoft to respond when commercial customer's businesses are at stake. Say what you want about Microsoft, i don't really like them, but their enterprise/commercial support is honestly pretty top notch, I'm really surprised it took them that long to get on top of that considering the calibre of customers involved.
I’m not claiming to be better at my job than their response team. I’m almost certainly not. The difference is I didn’t have to have meetings with 20 different internal stakeholders before acting.
About 20 years ago I was working with a large company that had dedicated servers in a huge east coast data center. The servers were firewalled, but they also were attached to a secondary network for scheduled backups.
I decided I should explore that further, and sure enough I was able to connect via SSH to other customers’ servers.
I alerted the hosting company, but they didn’t seem to take it very seriously.
Then we go back to his office, and load up the 1,000lb of computer gear he's donating and I'm transporting. We wheel 3 cartloads of stuff down the back elevator, through the secured DC floor we saw through glass earlier that i couldn't take a picture of, and out an open back door right near where i parked my truck. no further interaction with security, no badge waving, no questions asked.
> A clipboard and a confident wave will get you into any building in the world!
- Microsoft introduced Jupyter notebook support for Cosmos (who even asked for this?) and it was turned on for all customers automatically February 2021
- Security researchers find a way to break out of the container running the notebook and get long lived credentials to other people’s databases
- The credentials can be used off network. I assume with the public Azure API.
Don’t let your intern projects go into production without thorough security review.
This is my early and potentially slightly erroneous understanding of the situation:
- Jupiter Notebook support was enabled if you used the SQL API and not the Gremlin or MongoDB API. What is unclear is whether breaking out of the Notbook container gave you access to keys for just instances using SQL API or any API.
- CosmoDB is very weak in enforcing identity perimeters so keys are a weak point. Enforcing something like hourly key rotations is left up to the customer to build.
- Azure is pretty much an "it's public IP" unless you explicitly make it private. And even when they add controls like Private end points they have weird routing mechanisms that can result in traffic bypassing controls like hub firewalls.
- Using things like CosmoDB firewalls, private endpoints, automatic regular key rotations, and only enabling features you need should have helped if this was a hostile breach.
Take everything I said with a pinch of salt :)
Lots of users. The functionality itself is great. CosmosDB is a nice OLTP datastore with a flexible schema, and the ability to run OLAP queries over that data with SQL makes it a powerful tool.
Now, I bet you anything I could leave my current employer, get a job with a security firm and then find a major hole in my old company's service. Perhaps one that my team warned the CosmoDB team about internally. Or am I just being cynical?
If they still have that hole 18 months later, they kind of deserve it...
The way to avoid this, wouldn't have been firewalls, but using another database
Then mailing customers on the 26th.
Depending on whether or not this can be classified as a 'personal data breach' (IMO you should treat it as such, since it has such widespread implications) that may be a GDPR violation. The GDPR requires 72h notice to the data protection authorities and notice without 'undue delay' to the controller.
Will be interesting to learn more info down the road
Its been a long time since someone forced me to break out these articles. Very comforting to see the justification continue with such frequency.
Don't get me wrong - I absolutely love parts of what Microsoft does. Just not this whole cloud thing.
I get, on average, 3 Microsoft major change notifications every day. Can Microsoft even keep up with their own mess internally? I totally stopped trying from the outside. I just keep an eye on .NET and visual studio stack these days. Everything else winds up in my spam folder.
Then I read the article...
> The flaw was in a visualization tool called Jupyter Notebook, which has been available for years but was enabled by default in Cosmos beginning in February.
Azure actually, in my opinion, made the original Cosmos worse via "synergy"; but apparently they ALSO used the name for something unrelated :) WTF? Now I cannot even tell people anymore because the name is forever associated with this debacle.
