The world is composed of things. The things are grouped in sets. Numbers are equivalence classes of sets.
The set of the medium-sized planets of the Solar System is equivalent in number with the set of the big planets of the Solar System and also equivalent in number with the set of the big satellites of Jupiter (i.e. 4).
Such equivalences between sets of things exist regardless if there are any sentient beings that recognize those equivalences and there are circumstances when for instance the evolution in time of some sets of things is determined or influenced by the relationship between the numbers of things that compose each set.
By who?
If one take the position tha minds are not part of the "real world" then you end up defining "reality" by some random model.
It is just as absurd as saying that the only thing that exists is my own perception and you are "just" a ghost my mind is "thinking into existence".
It sounds less crazy only because we are used to reductionism being generally more useful, but what is the usefullness of concepts like reality and existence when defined to mean that we and our minds (the only thing we perceive) are not "real" or do not really "exist"?
The world is not made of a homogeneous substance, but there are various kinds of groupings at various levels, nucleons and electrons group in atoms, atoms group in molecules, molecules group in pebbles, stars group in galaxies and so on.
E.g. there are 10 rock formations in this part of the desert, and 2 rocks in the other side.
There is a clustering (set) of 10 vs 2 even if we aren't there to measure them and group them.
As are other sets co-existing (like the whole desert has 12 rock formations, while some other has only sand).
