There are valid criticisms of Docker/Podman isolation but it's not a binary "secure/not secure" thing, and honestly in this use case I don't see a major difference, apart from it being easier for a user to weaken the isolation provided by the container engine.
Docker/Podman security is essentially Linux security, it just uses namespaces+cgroups+capabilities+apparmor/SELinux+seccomp filters. There's a larger attack surface for kernel vulns when compared to VM hypervisors, but I've not heard of an LLM trying to break out by 0-day'ing the Linux kernel as yet :)