Why not just do remote model isolation? Like remote browser isolation. Run your local model / agent on a little box that has access to the internet and also has your repository, but doesn't have anything else. Like BrowserBox.

You interact with and drive the agent over a secure channel to your local machine, protected with this extra layer.

Is the source-code the secret you are trying to protect? Okay, no internet for you. Do you keep production secrets in your source-code? Okay, no programming permissions for you. ;)