While it does make more sense for them, that it not even clear as it seems. Library authors rarely define the public API because it is very tedious and hard to complete---there are only some sort of fuzzy and implicit "common sense" definitions. Whenever "breaking" changes happen the definition gets stronger (but still incomplete), and over the time it would encompass every observable aspect, as the Hyrum's Law suggests.

In either case semantic versioning is not tremendously useful because either users have an incomplete expectation of what major, minor and patch versions mean, or they will be notified of every possible change and the version distinction becomes useless. Semantic versioning is still useful because it was a codification of existing practice where the expectation can be good enough to avoid most issues. There is no actual value added by the codification in my opinion.