This logic is at the root of almost everyone who has been complicit in evil through history. It would once have been said about the abolition of slavery, gay rights, etc. It is a lazy attitude that's entirely self-justifying. Change is often opposed primarily by the great mass of people who use this same logic, in spite of them recognising that the change would be good.