I agree that changing tax law would be the best remedy but I'm not entirely sure if tax "avoidance" on a massive scale doesn't have some ethical considerations. When the people engaging in said avoidance have an outsized influence on the way tax laws are set up and interpreted relative to working/middle class citizens, I can certainly understand why some might attach an ethical consideration to such schemes. Just to be clear, this is an issue that I myself still haven't come to a conclusion on, but at the very least I can see both sides of the argument. I don't necessarily think it's "silly", perhaps naive or ignorant on my part though.
With regards to your second point, I'm not sure if I agree with your statement that "people with loads of money tend to be pretty prudent when ensuring what they are doing with their taxes is legal". I mean, I'm sure this is the case for many people but it seems that illegal tax evasion is a pretty serious global problem so clearly there are plenty of people who's main concern seems to be not getting caught rather than following the law. I also don't have a tremendous amount of faith that the US or other governments can adequately tackle the problem of enforcement at this time. Again, I could very well be wrong on all counts.
Mr. Soros in particular is problematic to me since he has a prior conviction for insider trading and there have been allegations surrounding his Quantum funds for many years. The secrecy and opaqueness with which he runs his investment vehicles certainly seems somewhat antithetical to an "open society", I wonder if the irony is lost on Mr. Soros.