For you it isn't 1/6 but for him it is. From your perspective can you derive from the knowledge you have why that for him the probability is 1/6?
And I mean derive as in derive the mathematical probability from all your knowledge about physics and the dice to explain why the lack of knowledge appears to him as a probability.
This is the essence of my question. We assume it. Probability is taken to be true for no reason, similar to how we just assume logic to be true.