I don't think it's that big of a deal; do you really care what the perception of "11 in the morning" is for someone 1,000 years ago? This kind of thing is pretty cultural anyway, and a slow drift over a thousands of years doesn't really matter.
The main reason we have the "new" Gregorian calendar is because of religious reasons, not because people were having huge practical problems with the old (slightly less accurate) Julian calendar.
Plus the current leap second system won't really deal with the long-term drift anyway because the earth's rotation keeps slowing, so in a few hundred years we'd need more leap seconds than the current system allows, and eventually we'd need a "leap second" every day because the day is a second longer (around the year 6000 IIRC).