Not disagreeing with you, but would like to offer a counter story. I personality live in what used to be an Olympic villag. Or, more accurately, was
supposed to be the Olympic village in the bid for Stockholm as the host city of the 2004 summer olympics. Interestingly, just like the Rio olympics the theme was environmental sustainability. The bid failed, but they went ahead with the plans anyway, and now it's lauded as a major success in urban redevelopment, taking what used to be a run down harbor and factory area which had mostly turned into a really shady part of town, and making it a great place for people to live and work. An old waste dump is now repurposed as a ski slope, and even hosted the Ski World Cup is past winter.
While not technically an Olympic development, the idea was always to build a village that could then serve as a new residential area, and goes to show that the olympics can very much be the necessary catalyst in getting major developments off the ground. If IOC wanted to, they could absolutely push for all sorts of change by forming host requirements thereafter. They may want to claim they're non-political, but I don't think that's even possible on the kind of scale they operate.