How we rooted Copilot (opens in new tab)

I bet the container was in an isolated VM too.

I would give the one engineer the credit for doing things better, not Microsoft. Microsoft overall culture of security is terrible. Look at the CISA report.

If they had found and reported a container breakout I expect they would've got a bug bounty from it!

Are there any known unfixed container breakouts at the moment in the kind of systems Microsoft are likely to be using here?

They're not. It's better to think of Copilot as a collaborative storytelling session with a text autocomplete system, which some other program is rudely hijacking to insert the result of running certain commands.

Sometimes the (completion randomly selected from the outputs of the) predictive text model goes "yes, and". Other times, it goes "no, because". As observed in the article, if it's autocompleting the result of many "yes, and"s, the story is probably going to have another "yes, and" next, but if a story starts off with a certain kind of demand, it's probably going to continue with a refusal.

Also shows you how shit an LLM is for defence, as it actively helps you look for exploits.

LLM is like an insane quadruple agent and you dont know whose side it is on (if any at all)

How do you figure? I don’t see anything that suggest they work for ms/azure?

This is very out of date. They now often trigger tooling and return the outputs of the tooling.

The bigger issue is around "material non‑public information" in stock market terms - things like unreported sales figures which someone could use to make trading decisions.

Using that information for trading is illegal, but so is exposing that information outside of approved channels.

This reminds me of that one time after working at a company for 4 months they informed me they were in a middle of an IP lawsuit which is part of the reason they hired me to rewrite the front end without knowing that was going on. That was f*(ked for reasons.

Whatever the case, the only time people look at your social media history is to look for attacks and the only reason they will look at a company's slack messages and emails are to look for attacks during discovery.

I would argue that company secrets are mostly useless for the company but very, very useful to other companies. For this reason, there should be retention policy of a day or two for almost all communication unless it is important, required by law, or documentation. And, definitely do not share that information with the public without good reason.

Except when they aren't. Defence in depth and zero trust and short expiry makes them way less useful for sure.

Startups are probably most vulnerable as they are likely to use more "pet" techniques for infra, like SSH open to any IP to make changes.

Then why are they secret?

That's why corporate espionage is a really lucrative industry?

Of course it depends what secrets. 99% will just be internal process drivel and inter departmental bickering but there's some real important stuff in there too.

I looked for an alleged case of an LLM apparently reproducing email signatures—but couldn’t find it exactly, and of course many email signatures have been published over the years, especially on newsgroups. (Maybe it was conspiratorial kind of thinking from web commenters assuming ChatGPT was training on emails users were feeding it, which as mentioned certainly doesn’t need to be the case.)

Something like the top screenshot here, though:

https://www.zdnet.com/article/chatgpt-can-leak-source-data-v...

(not parent commenter but) tl;dr no

  > a container breakout looked out of the question as every possible known breakout had been patched

This is the part that concerns me. It only encourages an attacker to sit on an exploit like this until a new container breakout is discovered.

Severity is based on impact. What was the impact here beyond single container and that specific user instance? Feels like moderate was okay, or even too high.

IMO if they truly don't consider it dangerous then they shouldn't have considered it a vulnerability at all, just a non-security bug. Labeling it a moderate vuln and not paying just seems like a bad middle ground to me, as though they haven't really decided if restricted root permissions is part of the security model or not.

Maybe this was their honeypot container.

It's still good for reputation. This is by a researcher at a company, so a benefit for both of them. Plus if we didn't have bug bounty programs, they'd have to willingly work at Microsoft to do this research.

Could say the same thing about open source software.

It mostly pays in career benefits. Same reason why plenty intern for free.

Well a lot of people do this kind of work to be able to commit crimes.

i don't think they did the work for them. they just reported it to them.

They didn't find anything they could do with it but that container isn't there for no reason. I agree with the rating but it's nonetheless worrying. You don't leave the house you bought unlocked because there's nothing in it to steal yet.