Wouldn't direnv just mean it will now send up your .envrc file? I think what would work even better is combining direnv with pass[0] so that if it does get uploaded, it will be encrypted. ie:
export SECRET_KEY=$(pass work/secret_key)
[0] https://www.passwordstore.org/