The Gradient between the gravitational forces on different parts of the earth is larger for the moon though, as you can see by the tides. I would assume this would have a bigger effect than the total force. But you are right, a comet or asteroid isn’t going to do anything.