"If it's the latter, what you're really struggling with is not the library, but the domain it's trying to represent."
But this doesn't change anything of the programmers problem. If I stumble on a bug in a physics libary I am using for a game, I cannot just jump right into there and go fixing things. I mean I can start doing it, but at the cost of not getting anything else done for quite some time.
There are lots of hard domains in programming. Cryptography is hard. Networking is hard. Fast rendering is hard. Efficient DBs are hard. OS are hard. Drivers are hard.
You can maybe fix a trivial error in such libaries, but everything else is usually a (big) project on its own.