It doesn't behave this way. If you're thinking of the DCT it uses that's mostly 4x4 which is not very large. As for motion analysis there are so many possible candidates (since it's on quarter-pixels) that it can't try all of them and very quickly starts trying to filter them out.
