Not in the USA, where the "first-sale doctrine" means once you buy a book, you can do whatever you want with that copy of the book (lend, rent, sell, destroy) without needing a license. Libraries in the USA definitely don't pay a fee beyond the purchase price of the book (or they can legally lend donated books etc). Copyright holders don't make any additional money from library lending.
I am not familiar with how it works in other countries, but I have heard something about there being such a fee.
(It's not quite true to say libraries have existed for "millenia" though, with regard to this issue. Mass produced printing hasn't in fact existed for millenia, libraries 1000 years ago had hand-copied manuscripts, probably mostly scrolls. The effect on "the market"? For whatever reason authors were writing then it was not to make money by selling reproductions of their writings, that wasn't a thing. Which means, yeah, btw, people still wrote things and made up stories even when they couldn't make money by charging people for copies to read...)