Yeah, my point is that whether you're changing a manifest and doing an apply versus a patch isn't relevant, the relevant bit is whether or not you have a version history and in both cases you do via your CD pipeline and via k8s deployment revision history. You can also commit to git, but I don't think there's much value in committing every image version change to git.
Also, working in raw manifests isn't a panacea; if you delete a resource's manifest, Kubernetes won't delete its corresponding resource (maybe this is what you were referring to with your bit about `kubectl delete -f`). You need something that diffs the manifests against the current state of the world and applies the requisite patches (something like Terraform) but this isn't related to how you change image versions.