it was also never their island, they lived there as employees only. their ancestors were brought to the uninhabited island by the French

it would be like claiming an oil rig or country estate as your family's property because you've had several generations of drillers/domestic staff employed by the owners

It’s a bit different when the territory is settled by non-indigenous people for a few generations, versus occupied, or just a few structures built and no one really lives there at all.