> I'd used LFS and move the ACLs to the file server, if using a single repo was a hard requirement
If you're putting a few large files in LFS, or maybe a couple sensitive files, I can understand and I'd say you're still using Git, but with some extensions.
If you're putting an entire sensitive subtree in LFS, I don't think you're really using Git any more, in the sense that many of your standard Git workflows will have to be different.