Don't get me wrong, the folks behind Copilot are clearly, without any doubt smart, creative, and capable. But then... None of these issues (reproducing licensed code ad verbatim, non-compiling code, getting semantics wrong, and now this) are 0.01% edge cases that take specialized knowledge to see or trigger. I remember some of them being called days ago in the initial HN thread by people who haven't had beta access.
I really wonder how this announcement/rollout looked like on the management side of things. Because a) these shortcomings must have been known beforehand and b) backlash from people who feel threatened for their jobs/"stolen" of their open source work was (I guess) foreseeable? I've already read calls to abandon GitHub for competitors; this can hardly have been an acceptable outcome here.
Nevertheless, Copilot is still one of the most innovative and interesting products I've seen in a while.
Hilariously, this results in stuff like Copilot getting released to great big legal problems. Only then does the C suite actually notice the project and get upset that it is a legal nightmare for them.
I think the real secret to winning in big tech is that your job is just to keep your head down and keep the money rolling in without causing headaches for higher ups. Increase sales, make customers happy enough to keep paying, maybe release a cool product. But more importantly, don’t cause a major outage, burn the PR team, or get caught up in a legal kerfuffle.
https://code.visualstudio.com/docs/getstarted/telemetry#:~:t....
Maybe something else still goes over the wire...
- implemented plagiarism detection to attribute code to its source (where possible), then present the result together with the link. This makes Copilot same with Googling your answer and then copy-pasting the code. You are fully responsible
- implemented some regexes to filter out secrets, or even better, change the secrets to random values in the training data
- implemented a robots.txt like system so people have a method to ban the Copilot spider from their code
If they did these things before release it would have been so much better. But they are simple fixes so I see no technical obstacle.
To rephrase my comment above: I don't want to blame the team behind Copilot for not getting everything right on the first try. Neither am I in a position to do so, nor would I want to live in a world where smart people aren't allowed to make mistakes.
What irritates me is that there are two possible scenarios here:
1) They knew about potential issues and decided to release it anyway (without at least addressing them verbally). 2) They didn't.
And frankly, I don't know which one I like less. Even though it's still a beta/preview, either option seems to signal a degree of negligence? that feels unnerving given the potential impact of such a system.
That being said, if we do live in scenario 1) than I am certain that better framing could have prevented the PR fallout that we're seeing right now (at least partially). IMHO, GitHub (the platform) is still a great product after all.
This one, for example, seems it should be pretty easy to fix. You could even make a hack that replaces ALL sufficiently long and sufficiently random strings with garbage/zeroes, at the point of recall. The difference from the case of regurgitating GPL sources is that the information that it looks like an API key can be deducedd from the output of copilot, so you don't need to track it through the system like you would with a system of attribution.
I have been on both sides of that, where I have had to point out obvious flaws in an idea to very smart people, and have had clearly obvious flaw pointed out to me in one of my idea's...
I think it is completely possible that some or even all of the issues co-pilot is facing were unknown at the time of release, even if they are obvious to some
If so, it means that you wouldn't find them by a search, but they would still be revealed by the A.I.
> What data has GitHub Copilot been trained on?
> GitHub Copilot is powered by OpenAI Codex, a new AI system created by OpenAI. It has been trained on a selection of English language and source code from publicly available sources, including code in public repositories on GitHub.
So it's not too far-fetched, that they'd train AI on private repos data too...
Any free text generation AI service has the same weakness, that you can't control what your model would spill out.
I know because I accidentally pushed a secret myself. Mistakes like that can happen super easily when it's a simple project and you've not setup all your things correctly.
Everyone knows it’s usually far easier to write code than to read code. Writing code is a nonlinear process: you don’t start from the first character and write everything out in one single pass. Instead, the logic of the code evolves nonlinearly—add a bit here, remove a bit there, restructure a bit over there. Good code is written such that it can be mostly understood in a single pass, but this is not always possible. For example, understanding function calls requires jumping around the code to where the function is defined (and often deeper down the stack). Understanding a conditional with multiple branches requires first reading all the conditional predicates before reading the code blocks they lead to.
Reading, on the other hand, is naturally a linear process. Understanding code requires reconstructing the nonlinear flow though it, and the nonlinear thought process used to write it in the first place. This is why constant communication between partners during pair programming is essential—if too much unexplained code gets dumped on a partner, figuring out how it works takes longer than just writing it themself.
Copilot is like pair programming with a completely incommunicative partner who can’t walk you through the code they just wrote. You therefore still have to review most of it manually, which takes much longer than writing it yourself in the first place.
However, I've worked with people who struggle to write in English without introducing random punctuation, and don't "see" (or care about) the text on the screen enough to go back and fix it.
I think Copilot will be a great benefit to the lazy programmer, who understands the semantics, but just can't be bothered to get the indentation or other syntax correct.
That said, a more sophisticated linter might be useful in catching non-idiomatic, but syntactically/stylistically valid code that would thus be flagged as “valid” by current linters’ simple automata.
To me, this is a huge part of modern-day development. It's not only about producing functionally correct code, but also code that integrates well and is semantically relevant to the broader context of the application itself.
That doesn't mean Copilot's input will have no value, but it just means that developers will generally need to refactor that code in a way consistent with the app they're building.
https://docs.github.com/en/code-security/secret-security/abo...
Doesn’t make sense to implement fixes on CoPilot.
But it's not generating the secrets, it's generating code that contain those secrets.
For a comparison point, afl fuzzer actually generates never-seen, valid JPEG files "out of thin air", see http://lcamtuf.blogspot.com/2014/11/pulling-jpegs-out-of-thi... and matching conversation https://news.ycombinator.com/item?id=8571879
It would be interesting to combine both approaches and (interrupted by the sudden singularity ;-).
Well if it wouldn't replicate or even randomly generate malicious code, it would imply that CoPilot would somehow be able to solve Halting Problem or - at the very least - understand intent and purpose of both its output and training material.
Keep in mind that the very definition of "malicious code" is highly subjective, plus the intent and purpose aren't necessarily encoded in the program itself. If the latter were the case, there would be no need for documentation, requirements or specs.
But there's another side to this - if you write code using co-pilot against a popular Api - and co-pilot gives you a valid key - and you access data or a system you aren't supposed to - would you be liable under the various draconian antighacker laws?
If you pick up a key card from the street, and enter someone's home - you'd be trespassing after all..
However, while the keys are then already leaked, you'd have to go search for them. Copilot suggests you use them in you editor. That is not quite the same imo.
It goes from deliberately searching and using leaked keys to having them handed to you without context. I feel it is a bit like finding an unlocked bike, if you take it, it is still stealing. But here there is a guy at the bike parking lets say that is handing out bikes to anyone passing by. Not the best analogy, but i think it covers my point ;)
But yes I get your point but I also believe people still need to apply some sense to what co pilot suggests.
The issue is that it's impossible to tell if it's done that or not short of googling each line it comes out with
Anyway I've determined my reactions are now based on disappointment rather than thought, a bias that should be factored in
I hate this belittling attitude that HN has towards projects in certain fields. It's like if I dismissed any progress in graphics as "just glorified triangle-drawing".
It won't change nothing. It's madlibs for code. The 90% of programmers who are mediocre will drown out the 9% who are competent and the 1% who are talented.
This thing is worthless at best, annoying at everything and terrifyingly capable of destroying every programmer's productivity at worst. But it will stay, it will grow because stupid execs will keep dreaming of replacing their engineering talent with this, and Microsoft will laugh all the way to the bank.
It's like expecting that an AI trained on Shakespeare novels would be able to "help" a writer write Shakespeare-like novels. Sure, they might get something that might fool some people, but are they a writer? I think software is a lot more like "writing" than it is like "building".
What mostly annoys me is that this is a win-win for github regardless of the outcome. If people buy into it (even for just a while, and it currently seems like some really smart people are buying into it) they'll carve a huge piece of a new market. If it fails, they'll make it seem like an experiment into the whole ethical gray area of what should and shouldn't be used for training, and that they just wanted to draw attention to it.
Copilot is a glorified Markov chain autocomplete sitting on a huge dump pile of data. It is not aware of constructs such as "licenses" or "secrets" most people would have expected from AI. To prevent it from spilling secrets everywhere, a developer ~~should teach the AI a concept of secrets and the meaning of licenses.~~ has to implement a filter. A regexp-based one will do, I guess.
Deep neural networks have stuck us in hope-fueled uncanny valley, and very smart people tend to become very confused about their technology when they're subject to it.
This technology has its place about heuristic programming, definitely, but is not AI.
All the API keys are still working and who knows, someone might complain about a huge fee right in here because they forgot to revoke it. Only time will tell.
I am certainly going to avoid this contraption. No thanks and most certainly no deal.
Downvoters: So are you saying GitHub Copilot DOES NOT do the following:
Leak working API keys in the editor.
Generate broken code AND give you the wrong implementation if you add a single typo?
Copy and regurgitates copyrighted code verbatim.
Guesses 1 out of 10 tries.
Send parts of your code when you type in the editor.
The content of your editor is sent by Copilot to its cloud service (the FAQ says: "The GitHub Copilot editor extension sends your comments and code to the GitHub Copilot service"). So yes any editor content is leaked, including sensitive information.
But is this content sent to other Copilot users? AFAIK, no. The FAQ mentions the OpenAI Codex as the training set.
They advertise an English-to-French translation service powered by AI. But it appears that nobody who is French native has even reviewed the service presentation. When the marketing material is just a joke, what can you expect in production if as a customer you use the service?
Just this example tells me much about the internal organization of the company.
> GitHub CEO acknowledges the issue... still waiting for them to pull the plug
I agree this is an issue for co-pilot as well - but it's really on send grid to invalidate keys that are known to be leaked?
Yes, that's inconvenient for the affected customers - otoh they won't get billed for other people's usage - or dinged for someone spamming using their keys...
I leaked one when I accidentally left one in a repo I was making public. Took 15 minutes for sendgrid to drop it after putting the repo public.
And it's another (bigger) issue for Copilot to pick up that API key and put it in someone else's project.
Copilot is an issue, but I would assume malicious agents are already doing copilot-like things, just not in the open.
They should scrap it and Microsoft should be ordered to sell Github because they have a conflict of interest.
For example Microsoft has access to your private repos and can do things like co pilot with your data. Who knows maybe your code powers Windows 11 now.
For example https://docs.github.com/en/code-security/secret-security/abo...
The fact that Copilot recreates API keys that still work makes me wonder if they come from a semi-public place, because SendGrid is usually quite fast at blocking API keys that were accidentally made public.
Just a quick search:
https://grep.app/search?q=%28secret%7Capi%29_%3Fkey%5Cs%3A%3...
Not saying putting keys in a private, but 3rd party hosted repo, is a terrific idea.
> COPILOT SECURITY BREACH
> SendGrid engineer reports API keys generated by the AI are not only valid but still functional.
> GitHub CEO acknowledges the issue... still waiting for them to pull the plug or make a comment. :popcorn:
Quoting https://twitter.com/pkell7/status/1411058236321681414