the diff goes up for everyone, not just you. you still have a relative advantage over all the other miners who have not yet reproduced your efficiency gains.
'relative difficulty' was never germane to the discussion. The point is that the system self-corrects for more efficient energy use by increasing its energy use. And its a lot of energy.