I am not sure you are getting it. If you have 100$ and I steal it from you, you can argue that I don't deserve the 100$ worth of work that I can buy from the society using it (you deserve it). But that is not the contract. The contract is anyone who has the 100$ is entitled for services that is 100$ worth from the society.
And society should not really concerned with who the actual owner is. Its end of the bargain is completed as soon as it does 100$ worth of work and accept the 100$ as payment.
So the billion $ a billionaire have means that society is indebted for that much amount to someone. And the burden to fulfill that commitment is on the society until that billion $ is exhausted from payment.