How do you measure success? Number of users? Number of posts and comments?
It is possible to have a small community with few people and less activity, that maintains a higher quality of content than a bigger community with more people.
The main danger of being small might be the threat of becoming an echo chamber because you are not exposed to as much variety in ideas. But even big communities sometimes do a “pretty good job at” (i.e. succumb to) becoming an echo chamber.