undefined | Better HN

0 pointssimonw7mo ago0 comments

I think the core of the whole problem is that if you have an LLM with access to tools and exposure to untrusted input, you should consider the author of that untrusted input to be have total control over the execution of those tools.

MCP is just a widely agreed upon abstraction over hooking an LLM up to some tools.

A significant potion of things people want to do with LLMs and with tools in general involve tasks where a malicious attacker taking control of those tools is a bad situation.

Is that what you mean by context hygiene? That end users need to assume that anything bad in the context can trigger unwanted actions, just like you shouldn't blindly copy and paste terminal commands from a web page into your shell (cough, curl https://.../install.sh | sh) or random chunks of JavaScript into the Firefox devtools console on Facebook.com ?

0 comments

tptacek7mo ago

On the first two paragraphs: we agree. (I just think that's both more obvious and less fundamental to the model than current writing on this suggests).

On the latter two paragraphs: my point is that there's nothing fundamental to the concept of an agent that requires you to mix untrusted content with sensitive tool calls. You can confine untrusted content to its own context window, and confine sensitive tool calls to "sandboxed" context windows; you can feed raw context from both to a third context window to summarize or synthesize; etc.

tehryanx7mo ago

Assuming you feed everything into another context to make safe, doesn't the problem just come with it? Why can't the LLM propagate misbehaviour into that stage?

tptacek7mo ago

The boundary between contexts is like the boundary between a POST argument in a web app and the database query it will drive. The point is, regardless of the fact that the system is making LLM calls, LLMs don't influence the code that decides what can and can't pass through the boundary between contexts; human-verified code does that.

1 more reply

simonwOP7mo ago

Right - that's more or less the idea behind https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ and the DeepMind CaMeL paper: https://simonwillison.net/2025/Apr/11/camel/

The challenge is that you have to implement really good taint tracking (as seen in old school Perl) - you need to make sure that the output of a model that was exposed to untrusted data never gets fed into some other model that has access potentially harmful tool calls.

I think that is possible to build, but I haven't seen any convincing implementation of the pattern yet. Hopefully soon!

tptacek7mo ago

So, we've surfaced a disagreement, because I don't think you need something like taint tracking. I think the security boundary between an LLM context that takes untrusted data (from, e.g., tickets) and a sensitive context (that can, e.g., make database queries) is essentially no different than the boundary between the GET/POST args in a web app and a SQL query.

It's not a trivial boundary, but it's one we have a very good handle on.

2 more replies

j / k navigate · click thread line to collapse

0 pointssimonw7mo ago0 comments

MCP is just a widely agreed upon abstraction over hooking an LLM up to some tools.

A significant potion of things people want to do with LLMs and with tools in general involve tasks where a malicious attacker taking control of those tools is a bad situation.

0 comments

tptacek7mo ago

On the first two paragraphs: we agree. (I just think that's both more obvious and less fundamental to the model than current writing on this suggests).

tehryanx7mo ago

Assuming you feed everything into another context to make safe, doesn't the problem just come with it? Why can't the LLM propagate misbehaviour into that stage?

tptacek7mo ago

1 more reply

simonwOP7mo ago

Right - that's more or less the idea behind https://simonwillison.net/2023/Apr/25/dual-llm-pattern/ and the DeepMind CaMeL paper: https://simonwillison.net/2025/Apr/11/camel/

I think that is possible to build, but I haven't seen any convincing implementation of the pattern yet. Hopefully soon!

tptacek7mo ago

It's not a trivial boundary, but it's one we have a very good handle on.

2 more replies

j / k navigate · click thread line to collapse