do package managers make promises that they only distribute code that's been audited to not pwn you? I'm not sure I see the difference if I decided I'm going to run someone's software whether I install it with sudo apt install vs sudo curl | bash
You are already trusting the maintainers of your distro by running Software they compiled, if you installed anything via the package manager. So it's about the number of people.
This only applies to software distributed by your distro. For something as novel as Ollama, I severely doubt it's made it into anything other than the most bleeding edge(Arch and co). You'll have to wait a few years to get it into mainline Debian, Ubuntu, Fedora, etc. and of course it will be at a set version.