So that could be one reason it takes longer to buffer.
256b/s AAC is well enough for it to be indistinguishable from lossless.