I can assure you: If you're asking this question, you are far from that point.
Nowadays you can host projects on services like aws, heroku, etc. and they have pretty good performance right out of the box. You will rarely reach a point where performance is a huge issue.
This may happen if you hit the HN front page, get covered by the NYTimes, etc. but even then, you could just spin up some dynos to handle it temporarily and think about if this is just a spike or an actual growth. In 99% of the cases it's just a spike and you'll be back to normal the next day.
If you DO reach the point where performance becomes important, that probably means you have enough users. And in the worst case you could even ask your users for help. But if you're doing this well, you can really just tweet for help and many people will try to help you.
TLDR: Don't optimize until you get to a point where it becomes painfully obvious that you MUST optimize.