You ask why it should work differently but then give a good reason why it should work differently: sometimes splitting by bytes is not the best unit.

And it actually does often work differently for PNG! PNG has a handful of preprocessing options for the pixels. So in filter mode 2, for example, deflate is encoding the difference between each pixel and the pixel above it. More or less.