Honestly, it's the money from the cleared land. Horrors of nature are the reason people
stay away. People move in only when those horrors occupy resources people think can be put to a better use.
Yes, it's often enough dumb, short-sighted, self-destructive selfish behavior, which I absolutely do not condone. However, horror or disgust alone are nowhere near enough to get people to engage in such behavior. At most it gets people to try - and sometimes succeed - to clear invasive species out of the gardens they already have.