That said, rent regulation (mostly rent stabilization vs. rent control, these days) does skew the market, creating a small number of lottery winners lucky to have below-market rents and driving up rents for everyone else competing for the unregulated inventory, regardless of their economic status.
The prices are pulled out of thin air. The new developments have to charge usurious rents to recoup their costs, but a lot of individual owners bought their building outright in the 80s for peanuts and just charge whatever they think they can get. Any old building in the LES or east village is basically a mint for the owner.
There may be a shortage of affordable, nice apartments in places where people want to live. Good luck finding a nice place for < $2000 in the LES. However, there is no shortage of expensive, crappy apartments in "uncool" parts of the city. In a weird twist of reality it's now easier to find a good deal in the UES than it is in the LES...
Fun fact: if an apartment is rented for less than $2000, it becomes subject to rent stabilization.
Many landlords would rather have an apartment go vacant than become subject to rent stabilization. Better to lose a year's worth of rent now than 20 years worth of rent in the future, not to mention loss of flexibility (i.e., you can't sell the apt without permission of the tenant).
If you are trying to move to a similar apartment that you are in, in the same neighborhood, for the same price, you will be told your apartment does not exist.
If you go to a management company trying to get into a big complex like StuyTown or any of the luxury rentals you will be quoted a price that is pulled out of thin air, and you will be lied to that "this is the last apartment available in XYZ highrise."
In short, managers are absolutely sitting on the best deals because they ALWAYS try to move the riskier product first. As the market gets worse, there is more risky property available, and thus it is even HARDER to find a good deal in a bad economy through brokers (the current gatekeepers of the market).
In 2008, you'd definitely have been right.