All it takes is one invited user to open the door to bots.
Also, if the bots are smart, they'll add real people too and take them down with them.
Your tree could for instance be pruned - you can still invite people, but the people you invited can no longer invite people.
There are not a lot of sites which have tried this and failed. Those which have tried to be even a little bit clever about it, have succeeded pretty well (Advogato was a really early example).
What there have been, are sites which rejected such restrictions after a while, because they would rather have a big number to show to investors than real people. Many have even run the fake accounts themselves (e.g. Reddit).
Invite only. You get a number of invites per year etc. And once a year an open door or so