Do people land on your website through search engines? If so, would it be wise to invest money for search engine optimisation?
Would I pay for SEO: No.
SEO is not the only way to get traffic to your site. Most of my traffic coming to my site comes from my blog, where I write 1-2 articles per month. I try to make the posts interesting and insightful to the type of crowd that would include my potential clients (startups).
An added benefit of blogging is that your name will start to get recognized as an expert in your field (assuming your articles are good), which will play in your favor when you're talking with prospects.
All the replies I got so far (both on HN and /r/freelance) seem to be negative.
Do you have an idea what makes a difference in your case? Is it your niche? Maybe, as you said, the fact that you establish yourself as an authority in your field through your blog?
Whenever I talk to a new prospect (potential client), I ask them how they found my site and what caught their attention. The answer to the first question is usually my blog or an HN comment (just like this one!). The answer to the second question varies from "your site tells a good story" to something specific such as my ability to push code.
In general, you'll probably find that most work comes from you reaching out to people pro-actively. At that point, they're definitely going to check out your website. If you can link them to your best work and give them some well-written blog articles to read, you'll definitely see the benefit in later conversations.
A portfolio is mainly useful as something you send to qualified prospects, not something you expect people to stumble upon and offer you business (anyone wasting time doing that probably can't afford to pay you anyway).
Think of your web site as your business card -- it's how people get in touch with you once they've already been introduced; it seldom works as the introduction itself.
To make your task easier - you can use tools like Leadify http://leadify.in etc.