Truthfully, why does anyone buy any particular asset expecting to do better than the market? The short answer, I think, is that they perceive the information and access that they have to be relatively great enough (than other market actors) to eek out an edge over the market. People's perceptions aren't always right, and the very definition of recessions (something that happened once or twice in the last "decade") is that entrepreneurs make a cluster of errors all at once.

As for the "why," I'm sure a particular's VC's perception of a startup investment falls favorably on their own risk-reward scale, and is perceived to be in line with the time horizon for their investment (whatever that may be). To believe them or not is up to you, and is the result of your own perceptions given the information available.

The real issue is that VC was over-capitalized in 1999-2000. More capital chasing the same investments bids up valuations and increases copycat competition, which both lower returns.

It takes a looong time to rebalance the supply of capital since PE/VC funds typically raise their next fund before the results of their current fund are realized. So the venture landscape was likely overcapitalized from 1999-2005, if not longer.

Has the balance come back or do we again have too much money chasing too few great companies? Given the choice, I prefer to have more capital in the system. More funding for entrepreneurs.