AI-Exploits: Repo of multiple unauthenticated RCEs in AI tools (opens in new tab)

Someone should build a poc. Ai doesnt know things other than what it’s ingested. So for such an attack to be successful you’d need to tilt the statistic towards problematic code. You’d need loads and loads of repositories but its definitely doable.

Indeed. I am thinking that one way to protect data and ensure its integrity is to somehow use agents trained on trusted sources to validate that the content is secure? For instance to detect “injections” of malicious or ill written code. Same for other types of content, but difficult.

Suppose someone magically creates thousands of repositories that write about a specific way of doing c pointers but all allow for buffer overflows, or sql queries with subtle ways to inject strings.

One way to defend is each data source that goes into training is to have an ai agent asses the input sources.

But even so it’s extremely difficult to catch convoluted attacks (ie when an exploit can be made upon meeting certain criteria).

Until then i’d consider any code written by an ai and unsupervised by a competent person as potentially tainted.

> Protect AI is the first company focused on the security of AI and ML Systems creating a new category we call MLSecOps.

alright i looked you up, congrats on your fundraising. is there like an OWASP top 10 vuln list for MLSecOps? does it differ between traditional ML apps and LLM apps?

> With the release of this repository, Protect AI hopes to demystify to the Information Security community what pratical attacks against AI/Machine Learning infrastructure look like in the real world and raise awareness to the amount of vulnerable components that currently exist in the AI/ML ecosystem. More vulnerabilities can be found here: November Vulnerability Report

pratical --> practical