Doing so is, in fact, the point of abolishing leap seconds.
Imagine that we have leap seconds, and the very worst case conceivable happens for an entire century. 100 summer leap seconds and 100 winter leap seconds, that's a total of just over three minutes by 2122.
Can we tolerate such a disruption to the lives of ordinary people? Try asking anybody who lives in a place with Daylight Saving or "Summer" Time. They have twenty times more disruption and that happens twice every single year.