Sandboxing is great, and stricter Authorization policies are great too, but with these kinds of software, my biggest fear (and that's why I am not trying them out now) is prompt injection.
It just seems unsolvable if you want the agent to do anything remotely useful