We already accept an error of half an hour (most timezone offsets are multiples of a whole hour) or more (many timezone boundaries are widened to align with state or country boundaries). A couple of minutes is nothing next to that, and if start getting too far, we can simply redefine the timezone offsets (which we already do twice a year in many places, and any device which might be moved across timezone boundaries already have to deal with that).
We don't really do that on a second accuracy though. We need to add a whole new day to the calendar every four years to make up the difference. Meanwhile we've added less than half a minute worth of leap seconds.