If you stick to the high level "Hey, take a photo for me" APIs, it isn't a problem. If you want to take advantage of hardware specific features, the iPhone camera system varies from generation to generation, and sometimes within a single generation. On top of this, the AV frameworks have a mountain of undefined behavior which easily breaks with iOS updates. There are tons of iOS and model specific code branches that have cropped up in Halide over the years.
That said, it doesn't hold a candle to Android fragmentation. It's a major reason we will not touch the platform.