There are severe restrictions on density and heavy parking requirements just to start. It’s much much more complex than that, but to simplify - people who lived in a given area want it to remain the same.
SF Bay Area is creating 3+ jobs per new unit built (any unit). So the gap is growing rather rapidly, resulting in demand far outpacing supply.
Most of the peninsula cities/towns do not want any growth and want to remain single family home neighborhoods. However, based on demand, they should probably evolve more into either denser row houses or apartment/townhome type.
The way to restrict this from happening is to require more land per sq.ft. of housing and require more parking per bedroom and guest parking for Multifamily housing. All this makes land use less efficient and therefore more expensive, deterring development.
