Because other miners are required to use a higher timestamp for their block to be valid, they have no ability to correct my outlier time. You have essentially allowed me to put the blocktime permanently forward by the allotted wiggle room, and also you have allowed me to consume all that space you allocated for miners who have clock skew.
It's overall a small attack, but highlights that even tiny decisions can have consequences that impact security. Bitcoin accounts for this by requiring that timestamps be greater than the median of the past 11 blocks, and that's enough to prevent one evil miner from forcing the block times forward for all blocks.