Otherwise, without being able to objectively measure this then it's all smoke and mirrors and the persons reach these positions through other means.
If that's the case, then taxing an appropriate amount is fair. There's also a precedent with Roosevelt introducing a tax that only really applied to Rockefeller