Except that you signed a contract when you bought the property.

Tourism.

I lived near a small historic town with many buildings standing since the 1850s. Tourism is a HUGE industry that brings in dollars to local businesses. It's in the town's interest to preserve that income, so they mandate color and style codes for new buildings as well as restoring older buildings. This is complemented with many folksy festivals and re-enactments as part of drawing in tourist dollars.

When people balk at these codes, and they do all the time, there are several other larger and modern cities nearby where they aren't restricted in any design sense.

There is no clear sharp line between the two; form has always been part of the function of architecture. What your house looks like affects the rest of your community; therefore there is a democratic process that gives them input.

I used to live near an old historic town with buildings standing since the 1850s. New people and businesses are moving in all the time and stuff needs to be built. The town uses color codes and other design elements to preserve both the older buildings as well as making newer buildings match the historic tone. This is explicitly done to promote and keep tourism flowing into the town, which is a HUGE part of their income.

If an industry or contractor balks at these ideas (and they do every now and again) there are several other larger modern cities a few miles away with access to the Interstate and train yards. These don't share the "historic preservation" codes of this little town.

If the town allowed a free-for-all on design it would wreck it's main source of income and likely cause decay over the years as tourism dropped off.

Some people love teal and pink. Having to look at or try to sell a house next to a monstrosity can be pretty horrible (or the neighbors with junker cars all over the lawn). Having been in a couple of regulated areas, I'm not much of a fan of the micromanagement that happens. However, having had my value/quality of living majorly degraded by an industrial operation moving in next door in unregulated BFE, the risk of living somewhere without rules is higher than the cost of compliance, at least for me.

The regulation on brick color mentioned above is at the city level, not federal or state. You have districts that enforce ascetic rules typically to preserve the look of a neighborhood.