You of course can argue that anything and everything affects interstate commerce, but ... in 1995 in Lopez the Court held that, no, there are truly local things, and laws not dealing with commerce (like the gun-free zone for schools act that was struck down at that time) lack authorization from the Constitution.
And this view was again reinforced in Morrision. Congress tried to use the Commerce clause to criminalize rape against women. The Court held that even though violence against women surely has some effect on the economy, that is very far from substantial. Even if viewed in the aggregate.
Our freedoms suffer some, sure, but the wheat (and other) market(s) got saner, which helped a lot of people, greatly expanding their freedoms. (On the other hand Wickard is used to regulate medical marijuana, which takes away some freedom.) But, but... it also allowed (and would allow) fixing healthcare, which was (would be) a big freedom improvent.